[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:55:23 dist-test-slave-3h5h krb5kdc[1520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:55:23 dist-test-slave-3h5h krb5kdc[1520](info): set up 2 sockets
Aug 07 02:55:23 dist-test-slave-3h5h krb5kdc[1520](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 02:55:25.187091 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.419s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:55:25 dist-test-slave-3h5h krb5kdc[1520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:55:25.398520 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41349
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:25.798553 1539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:25.799008 1539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:25.799172 1539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:25.806896 1539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:55:25.807058 1539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:25.807169 1539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:25.807307 1539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:55:25.807416 1539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:55:25.818300 1539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:25.820344 1539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:25.822346 1539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:25.846848 1548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:25.846967 1546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:25.849704 1545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:25.853006 1539 server_base.cc:1034] running on GCE node
I20260807 02:55:25.854193 1539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:25.856487 1539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:25.857914 1539 hybrid_clock.cc:648] HybridClock initialized: now 1786071325857757 us; error 142 us; skew 500 ppm
I20260807 02:55:25.858507 1539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:25 dist-test-slave-3h5h krb5kdc[1520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071325, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:25.862291 1539 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:55:25.864923 1539 webserver.cc:460] Webserver started at http://127.1.121.126:37057/ using document root <none> and password file <none>
I20260807 02:55:25.866169 1539 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:25.866326 1539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:25.866847 1539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:25.871080 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "b6cec225dd5040faac2b6242e45abcf9"
format_stamp: "Formatted at 2026-08-07 02:55:25 on dist-test-slave-3h5h"
server_key: "9a12228b78c807442428ba6b0d974556"
server_key_iv: "1022cc52b9f9d21df8cb7bc08a24b1a6"
server_key_version: "encryptionkey@0"
I20260807 02:55:25.872133 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "b6cec225dd5040faac2b6242e45abcf9"
format_stamp: "Formatted at 2026-08-07 02:55:25 on dist-test-slave-3h5h"
server_key: "9a12228b78c807442428ba6b0d974556"
server_key_iv: "1022cc52b9f9d21df8cb7bc08a24b1a6"
server_key_version: "encryptionkey@0"
I20260807 02:55:25.880199 1539 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260807 02:55:25.886235 1561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:25.888412 1539 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260807 02:55:25.888729 1539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "b6cec225dd5040faac2b6242e45abcf9"
format_stamp: "Formatted at 2026-08-07 02:55:25 on dist-test-slave-3h5h"
server_key: "9a12228b78c807442428ba6b0d974556"
server_key_iv: "1022cc52b9f9d21df8cb7bc08a24b1a6"
server_key_version: "encryptionkey@0"
I20260807 02:55:25.889045 1539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:25.914629 1539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:25.920147 1539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:25.920532 1539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:25.946239 1539 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:41349
I20260807 02:55:25.946293 1626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:41349 every 8 connection(s)
I20260807 02:55:25.948505 1539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:55:25.951681 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 1539
I20260807 02:55:25.952080 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:55:25.954408 1509 external_mini_cluster.cc:1428] Setting key b03808a152e22d6e0e02904127bd6f7c
I20260807 02:55:25.956079 1628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:25.972532 1628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9: Bootstrap starting.
I20260807 02:55:25.980283 1628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:25.982556 1628 log.cc:824] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:25.987754 1628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9: No bootstrap required, opened a new log
Aug 07 02:55:25 dist-test-slave-3h5h krb5kdc[1520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:25.999112 1628 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cec225dd5040faac2b6242e45abcf9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41349 } }
I20260807 02:55:25.999840 1628 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:26.000070 1628 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cec225dd5040faac2b6242e45abcf9, State: Initialized, Role: FOLLOWER
I20260807 02:55:26.001669 1628 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cec225dd5040faac2b6242e45abcf9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41349 } }
I20260807 02:55:26.002123 1628 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:55:26.002429 1628 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:55:26.002708 1628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:26.006129 1628 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cec225dd5040faac2b6242e45abcf9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41349 } }
I20260807 02:55:26.007741 1628 leader_election.cc:302] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6cec225dd5040faac2b6242e45abcf9; no voters:
I20260807 02:55:26.009013 1628 leader_election.cc:288] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:55:26.009315 1634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:26.010197 1634 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [term 1 LEADER]: Becoming Leader. State: Replica: b6cec225dd5040faac2b6242e45abcf9, State: Running, Role: LEADER
I20260807 02:55:26.011397 1634 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cec225dd5040faac2b6242e45abcf9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41349 } }
I20260807 02:55:26.011919 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:25.960396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.1.121.126:41349)
0807 02:55:25.961964 (+ 1568us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:25.961982 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:25.967032 (+ 5050us) server_negotiation.cc:407] Connection header received
0807 02:55:25.979736 (+ 12704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:25.979768 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:25.980677 (+ 909us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:25.981378 (+ 701us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:25.985395 (+ 4017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:25.988564 (+ 3169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:25.990409 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:25.991712 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:25.992020 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:25.998199 (+ 6179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:25.998273 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:25.998320 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:25.998401 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:26.003529 (+ 5128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:26.004449 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:26.004464 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:26.004472 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:26.004750 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:26.005273 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:26.005279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:26.005283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:26.006358 (+ 1075us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:26.007166 (+ 808us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:26.008017 (+ 851us) server_negotiation.cc:299] Negotiation successful
0807 02:55:26.008648 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":734,"threads_started":1}
I20260807 02:55:26.016902 1628 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:55:26.016989 1639 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6cec225dd5040faac2b6242e45abcf9. Latest consensus state: current_term: 1 leader_uuid: "b6cec225dd5040faac2b6242e45abcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cec225dd5040faac2b6242e45abcf9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41349 } } }
I20260807 02:55:26.017478 1639 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:26.018790 1644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:55:26.016906 1641 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6cec225dd5040faac2b6242e45abcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cec225dd5040faac2b6242e45abcf9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41349 } } }
I20260807 02:55:26.019268 1641 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:26.028939 1644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:55:26.046600 1644 catalog_manager.cc:1347] Generated new cluster ID: c558fcd755444bc48d57f55ea16b8a4a
I20260807 02:55:26.046764 1644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:55:26.061802 1644 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:55:26.063534 1644 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:55:26.073464 1644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6cec225dd5040faac2b6242e45abcf9: Generated new TSK 0
I20260807 02:55:26.074503 1644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:55:26.146318 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41349
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:26.543516 1674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:26.543825 1674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:26.544015 1674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:26.552560 1674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:26.552690 1674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:26.552910 1674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:55:26.563264 1674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41349
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:26.564718 1674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:26.566156 1674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:26.583559 1684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:26.583593 1688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:26.583612 1682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:26.585096 1674 server_base.cc:1034] running on GCE node
I20260807 02:55:26.585881 1674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:26.586912 1674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:26.588200 1674 hybrid_clock.cc:648] HybridClock initialized: now 1786071326588109 us; error 69 us; skew 500 ppm
I20260807 02:55:26.588563 1674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:26 dist-test-slave-3h5h krb5kdc[1520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071326, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:26.592083 1674 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:55:26.593869 1674 webserver.cc:460] Webserver started at http://127.1.121.65:38013/ using document root <none> and password file <none>
I20260807 02:55:26.594684 1674 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:26.594841 1674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:26.595184 1674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:26.598291 1674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "87987d98e1d0498fa2ee0f7fa0080d69"
format_stamp: "Formatted at 2026-08-07 02:55:26 on dist-test-slave-3h5h"
server_key: "a0a2326c2744b55b9c08105fc8d4bcc1"
server_key_iv: "c45644344f29f31e6e25f534a4710f0c"
server_key_version: "encryptionkey@0"
I20260807 02:55:26.599172 1674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "87987d98e1d0498fa2ee0f7fa0080d69"
format_stamp: "Formatted at 2026-08-07 02:55:26 on dist-test-slave-3h5h"
server_key: "a0a2326c2744b55b9c08105fc8d4bcc1"
server_key_iv: "c45644344f29f31e6e25f534a4710f0c"
server_key_version: "encryptionkey@0"
I20260807 02:55:26.604707 1674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 02:55:26.608649 1695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:26.610234 1674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:55:26.610460 1674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "87987d98e1d0498fa2ee0f7fa0080d69"
format_stamp: "Formatted at 2026-08-07 02:55:26 on dist-test-slave-3h5h"
server_key: "a0a2326c2744b55b9c08105fc8d4bcc1"
server_key_iv: "c45644344f29f31e6e25f534a4710f0c"
server_key_version: "encryptionkey@0"
I20260807 02:55:26.610731 1674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:26.665236 1674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:26.671000 1674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:26.671523 1674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:26.673059 1674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:26.674932 1674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:26.675055 1674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:26.675277 1674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:26.675365 1674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:26.731392 1674 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:36051
I20260807 02:55:26.731452 1872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:36051 every 8 connection(s)
I20260807 02:55:26.733032 1674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:55:26.743201 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 1674
I20260807 02:55:26.743491 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:55:26.744388 1509 external_mini_cluster.cc:1428] Setting key 8a8818460d6e9f71b6223a75e2fe96eb
Aug 07 02:55:26 dist-test-slave-3h5h krb5kdc[1520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071326, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:26.760687 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:26.737046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:50247 (local address 127.1.121.126:41349)
0807 02:55:26.737837 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:26.737845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:26.738842 (+ 997us) server_negotiation.cc:407] Connection header received
0807 02:55:26.741177 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:26.741182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:26.741260 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:26.741370 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:26.743593 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:26.745040 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:26.747309 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:26.748154 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:26.748333 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:26.752764 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:26.752774 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:26.752781 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:26.752867 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:26.756537 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:26.757779 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:26.757784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:26.757786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:26.758039 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:26.758617 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:26.758622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:26.758624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:26.758953 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:26.759091 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:26.760200 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0807 02:55:26.760382 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":437,"threads_started":1}
I20260807 02:55:26.762490 1877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:26.737807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41349 (local address 127.1.121.65:50247)
0807 02:55:26.738669 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:26.738712 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:26.740879 (+ 2167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:26.741620 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:26.741629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:26.742390 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:26.743218 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:26.743246 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:26.745217 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:26.745221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:26.747150 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:26.747164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:26.748469 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:26.748475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:26.748762 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:26.749612 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:26.749639 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:26.752485 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:26.756744 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:26.756757 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:26.756778 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:26.757614 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:26.758178 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:26.758186 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:26.758192 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:26.758460 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:26.759108 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:26.759119 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:26.759687 (+ 568us) client_negotiation.cc:769] Sending connection context
0807 02:55:26.760076 (+ 389us) client_negotiation.cc:240] Negotiation successful
0807 02:55:26.760729 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":340,"threads_started":1}
I20260807 02:55:26.764506 1873 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41349
I20260807 02:55:26.765064 1873 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:26.766117 1873 heartbeater.cc:511] Master 127.1.121.126:41349 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
I20260807 02:55:26.769970 1583 ts_manager.cc:194] Registered new tserver with Master: 87987d98e1d0498fa2ee0f7fa0080d69 (127.1.121.65:36051)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
I20260807 02:55:26.772614 1583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:50247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:55:26.800920 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41349
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:27.199448 1880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:27.199741 1880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:27.199949 1880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:27.207572 1880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:27.207683 1880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:27.207891 1880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:55:27.218427 1880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41349
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:27.219841 1880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:27.221320 1880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:27.236698 1885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:27.236974 1886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:27.237789 1888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:27.238422 1880 server_base.cc:1034] running on GCE node
I20260807 02:55:27.239145 1880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:27.240228 1880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:27.241490 1880 hybrid_clock.cc:648] HybridClock initialized: now 1786071327241439 us; error 63 us; skew 500 ppm
I20260807 02:55:27.241839 1880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:27 dist-test-slave-3h5h krb5kdc[1520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:27.245330 1880 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:55:27.247025 1880 webserver.cc:460] Webserver started at http://127.1.121.66:40627/ using document root <none> and password file <none>
I20260807 02:55:27.247742 1880 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:27.247888 1880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:27.248224 1880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:27.251317 1880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "bd2f4761fb9542398a7b052b5d3f0886"
format_stamp: "Formatted at 2026-08-07 02:55:27 on dist-test-slave-3h5h"
server_key: "8bedd20d983a96e7198f7a1b1723ce85"
server_key_iv: "e476b5d3cba529d41ae3dbc909a8110a"
server_key_version: "encryptionkey@0"
I20260807 02:55:27.252100 1880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "bd2f4761fb9542398a7b052b5d3f0886"
format_stamp: "Formatted at 2026-08-07 02:55:27 on dist-test-slave-3h5h"
server_key: "8bedd20d983a96e7198f7a1b1723ce85"
server_key_iv: "e476b5d3cba529d41ae3dbc909a8110a"
server_key_version: "encryptionkey@0"
I20260807 02:55:27.258200 1880 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260807 02:55:27.262326 1900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:27.263926 1880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:55:27.264137 1880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "bd2f4761fb9542398a7b052b5d3f0886"
format_stamp: "Formatted at 2026-08-07 02:55:27 on dist-test-slave-3h5h"
server_key: "8bedd20d983a96e7198f7a1b1723ce85"
server_key_iv: "e476b5d3cba529d41ae3dbc909a8110a"
server_key_version: "encryptionkey@0"
I20260807 02:55:27.264386 1880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:27.283958 1880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:27.288933 1880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:27.289270 1880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:27.290524 1880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:27.292160 1880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:27.292265 1880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:27.292440 1880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:27.292516 1880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:27.335734 1880 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:37531
I20260807 02:55:27.335757 2070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:37531 every 8 connection(s)
I20260807 02:55:27.337654 1880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:55:27.343407 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 1880
I20260807 02:55:27.343660 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:55:27.344177 1509 external_mini_cluster.cc:1428] Setting key a1c7f827b210bccd33a550313d09e4af
Aug 07 02:55:27 dist-test-slave-3h5h krb5kdc[1520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:27.362965 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:27.341022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:43291 (local address 127.1.121.126:41349)
0807 02:55:27.341567 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:27.341572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:27.342609 (+ 1037us) server_negotiation.cc:407] Connection header received
0807 02:55:27.344670 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:27.344676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:27.344777 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:27.344960 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:27.346918 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:27.348193 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:27.349909 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:27.350920 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:27.351230 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:27.355744 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:27.355750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:27.355753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:27.355790 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:27.359054 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:27.360204 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:27.360208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:27.360211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:27.360362 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:27.360948 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:27.360952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:27.360954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:27.361249 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:27.361372 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:27.362389 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0807 02:55:27.362587 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":275,"threads_started":1}
I20260807 02:55:27.364941 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:27.341638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41349 (local address 127.1.121.66:43291)
0807 02:55:27.342439 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:27.342479 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:27.344386 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:27.345231 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:27.345239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:27.345975 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:27.346743 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:27.346763 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:27.348344 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:27.348348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:27.349711 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:27.349724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:27.351252 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:27.351256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:27.351460 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:27.352207 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:27.352237 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:27.355399 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:27.359233 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:27.359246 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:27.359268 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:27.360050 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:27.360481 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:27.360488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:27.360495 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:27.360805 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:27.361382 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:27.361393 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:27.361967 (+ 574us) client_negotiation.cc:769] Sending connection context
0807 02:55:27.362292 (+ 325us) client_negotiation.cc:240] Negotiation successful
0807 02:55:27.362934 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":349,"threads_started":1}
I20260807 02:55:27.366744 2073 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41349
I20260807 02:55:27.367080 2073 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:27.367986 2073 heartbeater.cc:511] Master 127.1.121.126:41349 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
I20260807 02:55:27.369827 1583 ts_manager.cc:194] Registered new tserver with Master: bd2f4761fb9542398a7b052b5d3f0886 (127.1.121.66:37531)
I20260807 02:55:27.370780 1583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:43291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:55:27.400576 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41349
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:55:27.777294 1873 heartbeater.cc:503] Master 127.1.121.126:41349 was elected leader, sending a full tablet report...
W20260807 02:55:27.804522 2081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:27.804858 2081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:27.804971 2081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:27.813504 2081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:27.813624 2081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:27.813809 2081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:55:27.825855 2081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41349
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:27.827464 2081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:27.829066 2081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:27.844928 2089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:27.844887 2093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:27.845000 2090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:27.846050 2081 server_base.cc:1034] running on GCE node
I20260807 02:55:27.846669 2081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:27.847705 2081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:27.848948 2081 hybrid_clock.cc:648] HybridClock initialized: now 1786071327848899 us; error 51 us; skew 500 ppm
I20260807 02:55:27.849438 2081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:27 dist-test-slave-3h5h krb5kdc[1520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:27.853369 2081 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:55:27.855310 2081 webserver.cc:460] Webserver started at http://127.1.121.67:40043/ using document root <none> and password file <none>
I20260807 02:55:27.856117 2081 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:27.856287 2081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:27.856662 2081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:27.860190 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "7351fabc078047a9beb53432247394a4"
format_stamp: "Formatted at 2026-08-07 02:55:27 on dist-test-slave-3h5h"
server_key: "b336b160f04b96944294a3421419c1ef"
server_key_iv: "84ff7a6cc7ec1aeaa4ac5f2336401d77"
server_key_version: "encryptionkey@0"
I20260807 02:55:27.861192 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "7351fabc078047a9beb53432247394a4"
format_stamp: "Formatted at 2026-08-07 02:55:27 on dist-test-slave-3h5h"
server_key: "b336b160f04b96944294a3421419c1ef"
server_key_iv: "84ff7a6cc7ec1aeaa4ac5f2336401d77"
server_key_version: "encryptionkey@0"
I20260807 02:55:27.867563 2081 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 02:55:27.871934 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:27.873880 2081 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:55:27.874078 2081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "7351fabc078047a9beb53432247394a4"
format_stamp: "Formatted at 2026-08-07 02:55:27 on dist-test-slave-3h5h"
server_key: "b336b160f04b96944294a3421419c1ef"
server_key_iv: "84ff7a6cc7ec1aeaa4ac5f2336401d77"
server_key_version: "encryptionkey@0"
I20260807 02:55:27.874331 2081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:27.887444 2081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:27.891687 2081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:27.891956 2081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:27.893184 2081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:27.894915 2081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:27.894989 2081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:27.895074 2081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:27.895119 2081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:27.940908 2081 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:41073
I20260807 02:55:27.941176 2270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:41073 every 8 connection(s)
I20260807 02:55:27.942310 2081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:55:27.944432 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 2081
I20260807 02:55:27.944674 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:55:27.945266 1509 external_mini_cluster.cc:1428] Setting key 991c9b4ada61bcbe68be89683e33ebc5
Aug 07 02:55:27 dist-test-slave-3h5h krb5kdc[1520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:27.970451 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:27.945435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:60587 (local address 127.1.121.126:41349)
0807 02:55:27.953312 (+ 7877us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:27.953317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:27.953336 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:55:27.954114 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:27.954119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:27.954194 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:27.954310 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:27.956362 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:27.957526 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:27.959191 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:27.959702 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:27.959819 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:27.964172 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:27.964178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:27.964182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:27.964218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:27.967010 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:27.967833 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:27.967838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:27.967841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:27.967966 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:27.968546 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:27.968550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:27.968553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:27.968929 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:27.969161 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:27.969824 (+ 663us) server_negotiation.cc:299] Negotiation successful
0807 02:55:27.970087 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7774,"thread_start_us":7586,"threads_started":1}
I20260807 02:55:27.971455 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:27.946341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41349 (local address 127.1.121.67:60587)
0807 02:55:27.950812 (+ 4471us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:27.950861 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:27.953676 (+ 2815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:27.954605 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:27.954618 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:27.955375 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:27.956157 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:27.956181 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:27.957708 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:27.957714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:27.959022 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:27.959035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:27.959931 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:27.959937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:27.960188 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:27.961018 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:27.961043 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:27.963902 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:27.967158 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:27.967165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:27.967180 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:27.967719 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:27.968213 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:27.968217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:27.968219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:27.968447 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:27.969094 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:27.969100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:27.969484 (+ 384us) client_negotiation.cc:769] Sending connection context
0807 02:55:27.969734 (+ 250us) client_negotiation.cc:240] Negotiation successful
0807 02:55:27.970249 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4110,"thread_start_us":3803,"threads_started":1}
I20260807 02:55:27.972954 2272 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41349
I20260807 02:55:27.973372 2272 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:27.974558 2272 heartbeater.cc:511] Master 127.1.121.126:41349 requested a full tablet report, sending...
I20260807 02:55:27.976305 1583 ts_manager.cc:194] Registered new tserver with Master: 7351fabc078047a9beb53432247394a4 (127.1.121.67:41073)
I20260807 02:55:27.977257 1583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:60587
I20260807 02:55:27.988006 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:55:28 dist-test-slave-3h5h krb5kdc[1520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:55:28 dist-test-slave-3h5h krb5kdc[1520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:55:28.045504 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:28.028726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.1.121.65:36051)
0807 02:55:28.029455 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:28.029468 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:28.029547 (+ 79us) server_negotiation.cc:407] Connection header received
0807 02:55:28.029829 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:28.029838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:28.030065 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:28.030240 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:28.032033 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.033107 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.034449 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.035092 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.035247 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:28.039818 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:28.039830 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:28.039845 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:28.039885 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:28.042877 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.043555 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.043562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.043564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.043704 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.044058 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.044061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.044063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.044403 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:28.044658 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:28.045006 (+ 348us) server_negotiation.cc:299] Negotiation successful
0807 02:55:28.045215 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":345,"threads_started":1}
W20260807 02:55:28.047040 1744 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59102
Aug 07 02:55:28 dist-test-slave-3h5h krb5kdc[1520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:55:28 dist-test-slave-3h5h krb5kdc[1520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:55:28.108204 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:28.092765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.1.121.65:36051)
0807 02:55:28.093087 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:28.093095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:28.093406 (+ 311us) server_negotiation.cc:407] Connection header received
0807 02:55:28.093577 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:28.093582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:28.093670 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:28.093779 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:28.095325 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.096123 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.097480 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.097993 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.098140 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:28.102354 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:28.102360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:28.102363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:28.102405 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:28.105528 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.106174 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.106178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.106181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.106331 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.106648 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.106652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.106654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.106913 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:28.107207 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:28.107539 (+ 332us) server_negotiation.cc:299] Negotiation successful
0807 02:55:28.107781 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260807 02:55:28.110095 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 1674
I20260807 02:55:28.124048 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:28.111162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.1.121.65:36051)
0807 02:55:28.111328 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:28.111332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:28.111744 (+ 412us) server_negotiation.cc:407] Connection header received
0807 02:55:28.111832 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:28.111836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:28.111897 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:28.111987 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:28.113306 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.114157 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.115546 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.116162 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.116274 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:28.118729 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:28.118734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:28.118738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:28.118770 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:28.121577 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.122185 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.122189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.122191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.122332 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.122766 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.122769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.122771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.123024 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:28.123195 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:28.123530 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 02:55:28.123688 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:55:28.125231 1868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:28.215149 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 1674
I20260807 02:55:28.239104 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 1880
Aug 07 02:55:28 dist-test-slave-3h5h krb5kdc[1520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:55:28.255578 2293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:28.239953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.1.121.66:37531)
0807 02:55:28.240563 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:28.240570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:28.240596 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:55:28.240672 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:28.240679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:28.240928 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:28.241063 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:28.242263 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.243293 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.244508 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.245111 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.245212 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:28.249737 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:28.249749 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:28.249764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:28.249801 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:28.252714 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.253612 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.253620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.253622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.253764 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.254192 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.254195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.254197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.254529 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:28.254787 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:28.255074 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 02:55:28.255280 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":352,"threads_started":1}
I20260807 02:55:28.257045 2066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:28.346177 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 1880
I20260807 02:55:28.369988 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 2081
Aug 07 02:55:28 dist-test-slave-3h5h krb5kdc[1520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:55:28.388895 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:28.371382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.1.121.67:41073)
0807 02:55:28.372017 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:28.372027 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:28.372068 (+ 41us) server_negotiation.cc:407] Connection header received
0807 02:55:28.372142 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:28.372165 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:28.372401 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:28.372640 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:28.374062 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.375362 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.376783 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:28.377743 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:28.377886 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:28.383011 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:28.383024 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:28.383039 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:28.383087 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:28.386183 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.386866 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.386873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.386875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.386995 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:28.387384 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:28.387388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:28.387390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:28.387733 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:28.387968 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:28.388236 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 02:55:28.388427 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":244,"threads_started":1}
I20260807 02:55:28.390163 2263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:28.485352 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 2081
I20260807 02:55:28.511519 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 1539
I20260807 02:55:28.512818 1621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:28.624428 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 1539
2026-08-07T02:55:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4923 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:55:28 dist-test-slave-3h5h krb5kdc[2303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:55:28 dist-test-slave-3h5h krb5kdc[2303](info): set up 2 sockets
Aug 07 02:55:28 dist-test-slave-3h5h krb5kdc[2303](info): commencing operation
krb5kdc: starting...
W20260807 02:55:29.892325 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:55:30 dist-test-slave-3h5h krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:55:30.110769 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:35425
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:30.459381 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:30.459676 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:30.459771 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:30.466993 2327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:55:30.467106 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:30.467160 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:30.467218 2327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:55:30.467262 2327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:55:30.478790 2327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:35425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:30.480228 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:30.481662 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:30.494103 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:30.494252 2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:30.494653 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:30.495198 2327 server_base.cc:1034] running on GCE node
I20260807 02:55:30.495977 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:30.497558 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:30.498778 2327 hybrid_clock.cc:648] HybridClock initialized: now 1786071330498730 us; error 31 us; skew 500 ppm
I20260807 02:55:30.499202 2327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:30 dist-test-slave-3h5h krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071330, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:30.508455 2327 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:55:30.510349 2327 webserver.cc:460] Webserver started at http://127.1.121.126:37269/ using document root <none> and password file <none>
I20260807 02:55:30.511196 2327 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:30.511340 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:30.511708 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:30.515162 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e"
format_stamp: "Formatted at 2026-08-07 02:55:30 on dist-test-slave-3h5h"
server_key: "1c9712a79f7b18bc106ac6720a3e0d9b"
server_key_iv: "2156240804f0990450368f716728452a"
server_key_version: "encryptionkey@0"
I20260807 02:55:30.516037 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e"
format_stamp: "Formatted at 2026-08-07 02:55:30 on dist-test-slave-3h5h"
server_key: "1c9712a79f7b18bc106ac6720a3e0d9b"
server_key_iv: "2156240804f0990450368f716728452a"
server_key_version: "encryptionkey@0"
I20260807 02:55:30.521857 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 02:55:30.525763 2351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:30.527078 2327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:55:30.527303 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e"
format_stamp: "Formatted at 2026-08-07 02:55:30 on dist-test-slave-3h5h"
server_key: "1c9712a79f7b18bc106ac6720a3e0d9b"
server_key_iv: "2156240804f0990450368f716728452a"
server_key_version: "encryptionkey@0"
I20260807 02:55:30.527520 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:30.543994 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:30.548095 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:30.548390 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:30.567402 2327 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:35425
I20260807 02:55:30.567422 2428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:35425 every 8 connection(s)
I20260807 02:55:30.568823 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:55:30.572206 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 2327
I20260807 02:55:30.572439 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:55:30.572715 2429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:30.573217 1509 external_mini_cluster.cc:1428] Setting key 36bd388db55132963a40ec58201427b1
I20260807 02:55:30.587393 2429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e: Bootstrap starting.
Aug 07 02:55:30 dist-test-slave-3h5h krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:30.590894 2429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:30.592108 2429 log.cc:824] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:30.594956 2429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e: No bootstrap required, opened a new log
I20260807 02:55:30.597472 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:30.575155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.1.121.126:35425)
0807 02:55:30.576032 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:30.576043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:30.576088 (+ 45us) server_negotiation.cc:407] Connection header received
0807 02:55:30.577767 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:30.577813 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:30.578208 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:30.578553 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:30.580492 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:30.581977 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:30.583816 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:30.584592 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:30.584741 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:30.589619 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:30.589627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:30.589641 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:30.589683 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:30.593313 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:30.594001 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:30.594009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:30.594023 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:30.594187 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:30.594553 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:30.594558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:30.594561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:30.595114 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:30.595444 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:30.595934 (+ 490us) server_negotiation.cc:299] Negotiation successful
0807 02:55:30.596340 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":422,"threads_started":1}
I20260807 02:55:30.602901 2429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35425 } }
I20260807 02:55:30.603288 2429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:30.603394 2429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1473e715044c9bbc6b2b8ba9ddd29e, State: Initialized, Role: FOLLOWER
I20260807 02:55:30.603925 2429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35425 } }
I20260807 02:55:30.604173 2429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:55:30.604313 2429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:55:30.604485 2429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:30.607110 2429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35425 } }
I20260807 02:55:30.607693 2429 leader_election.cc:302] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d1473e715044c9bbc6b2b8ba9ddd29e; no voters:
I20260807 02:55:30.608309 2429 leader_election.cc:288] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:55:30.608521 2435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:30.609398 2435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [term 1 LEADER]: Becoming Leader. State: Replica: 4d1473e715044c9bbc6b2b8ba9ddd29e, State: Running, Role: LEADER
I20260807 02:55:30.609977 2435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35425 } }
I20260807 02:55:30.610450 2429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:55:30.613384 2437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d1473e715044c9bbc6b2b8ba9ddd29e. Latest consensus state: current_term: 1 leader_uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35425 } } }
I20260807 02:55:30.613826 2437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:30.614514 2436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1473e715044c9bbc6b2b8ba9ddd29e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35425 } } }
I20260807 02:55:30.614920 2436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:30.614981 2445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:55:30.623822 2445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:55:30.635004 2445 catalog_manager.cc:1347] Generated new cluster ID: f8c090f138b449a49f25fddf7f5200fb
I20260807 02:55:30.635136 2445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:55:30.655805 2445 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:55:30.657930 2445 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:55:30.673269 2445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d1473e715044c9bbc6b2b8ba9ddd29e: Generated new TSK 0
I20260807 02:55:30.674487 2445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:55:30.734757 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35425
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:31.086795 2465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:31.087117 2465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:31.087280 2465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:31.094830 2465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:31.094942 2465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:31.095134 2465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:55:31.105173 2465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:31.106590 2465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:31.107944 2465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:31.121905 2474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:31.121860 2471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:31.121917 2472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:31.123028 2465 server_base.cc:1034] running on GCE node
I20260807 02:55:31.123647 2465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:31.124596 2465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:31.125800 2465 hybrid_clock.cc:648] HybridClock initialized: now 1786071331125726 us; error 58 us; skew 500 ppm
I20260807 02:55:31.126161 2465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:31 dist-test-slave-3h5h krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:31.129433 2465 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:55:31.130858 2465 webserver.cc:460] Webserver started at http://127.1.121.65:35465/ using document root <none> and password file <none>
I20260807 02:55:31.131585 2465 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:31.131729 2465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:31.132040 2465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:31.135010 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "b7366fa4c67946c394f12d35858460f4"
format_stamp: "Formatted at 2026-08-07 02:55:31 on dist-test-slave-3h5h"
server_key: "dcbb47bf3ddff44edc9612446b006ebf"
server_key_iv: "386355f415633398b6a4107c735e90db"
server_key_version: "encryptionkey@0"
I20260807 02:55:31.135816 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "b7366fa4c67946c394f12d35858460f4"
format_stamp: "Formatted at 2026-08-07 02:55:31 on dist-test-slave-3h5h"
server_key: "dcbb47bf3ddff44edc9612446b006ebf"
server_key_iv: "386355f415633398b6a4107c735e90db"
server_key_version: "encryptionkey@0"
I20260807 02:55:31.141181 2465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 02:55:31.144708 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:31.146199 2465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:55:31.146386 2465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "b7366fa4c67946c394f12d35858460f4"
format_stamp: "Formatted at 2026-08-07 02:55:31 on dist-test-slave-3h5h"
server_key: "dcbb47bf3ddff44edc9612446b006ebf"
server_key_iv: "386355f415633398b6a4107c735e90db"
server_key_version: "encryptionkey@0"
I20260807 02:55:31.146603 2465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:31.164793 2465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:31.168779 2465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:31.169102 2465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:31.170099 2465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:31.171485 2465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:31.171586 2465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:31.171739 2465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:31.171818 2465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:31.208384 2465 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:35955
I20260807 02:55:31.208416 2647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:35955 every 8 connection(s)
I20260807 02:55:31.209880 2465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:55:31.217798 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 2465
I20260807 02:55:31.218055 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:55:31.218796 1509 external_mini_cluster.cc:1428] Setting key f6916d9517f5de64f6bc386e412a4495
Aug 07 02:55:31 dist-test-slave-3h5h krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:31.231526 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:31.212924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:50559 (local address 127.1.121.126:35425)
0807 02:55:31.213396 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:31.213400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:31.214426 (+ 1026us) server_negotiation.cc:407] Connection header received
0807 02:55:31.216298 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:31.216304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:31.216397 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:31.216524 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:31.218127 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:31.218953 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:31.220477 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:31.220956 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:31.221087 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:31.224529 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:31.224534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:31.224538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:31.224569 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:31.227615 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:31.228764 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:31.228769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:31.228771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:31.228957 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:31.229437 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:31.229445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:31.229451 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:31.229834 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:31.230060 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:31.230741 (+ 681us) server_negotiation.cc:299] Negotiation successful
0807 02:55:31.231022 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260807 02:55:31.232596 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:31.213487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:35425 (local address 127.1.121.65:50559)
0807 02:55:31.214268 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:31.214307 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:31.216017 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:31.216754 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:31.216762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:31.217345 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:31.217947 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:31.217973 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:31.219120 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:31.219124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:31.220327 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:31.220341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:31.221183 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:31.221193 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:31.221416 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:31.222089 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:31.222118 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:31.224295 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:31.227770 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:31.227784 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:31.227805 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:31.228609 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:31.229093 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:31.229099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:31.229103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:31.229322 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:31.229967 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:31.229974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:31.230320 (+ 346us) client_negotiation.cc:769] Sending connection context
0807 02:55:31.230585 (+ 265us) client_negotiation.cc:240] Negotiation successful
0807 02:55:31.231079 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":296,"threads_started":1}
I20260807 02:55:31.234345 2648 heartbeater.cc:347] Connected to a master server at 127.1.121.126:35425
I20260807 02:55:31.234705 2648 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:31.235685 2648 heartbeater.cc:511] Master 127.1.121.126:35425 requested a full tablet report, sending...
I20260807 02:55:31.238150 2381 ts_manager.cc:194] Registered new tserver with Master: b7366fa4c67946c394f12d35858460f4 (127.1.121.65:35955)
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
I20260807 02:55:31.239773 2381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:50559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:55:31.268170 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35425
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:31.638736 2659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:31.639111 2659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:31.639333 2659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:31.647958 2659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:31.648129 2659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:31.648355 2659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:55:31.660163 2659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:31.662259 2659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:31.663807 2659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:31.678836 2670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:31.678886 2675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:31.678920 2671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:31.680305 2659 server_base.cc:1034] running on GCE node
I20260807 02:55:31.681123 2659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:31.682211 2659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:31.683486 2659 hybrid_clock.cc:648] HybridClock initialized: now 1786071331683418 us; error 40 us; skew 500 ppm
I20260807 02:55:31.683853 2659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:31 dist-test-slave-3h5h krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:31.687537 2659 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:55:31.689342 2659 webserver.cc:460] Webserver started at http://127.1.121.66:38359/ using document root <none> and password file <none>
I20260807 02:55:31.690160 2659 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:31.690344 2659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:31.690807 2659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:31.694314 2659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "a8d4c1899a2947de86b5fa41021e6d3e"
format_stamp: "Formatted at 2026-08-07 02:55:31 on dist-test-slave-3h5h"
server_key: "64f614b5d07a3104c5ef64db7ddcdf67"
server_key_iv: "9b77e8441816ac2f53e3d5507eeb02ab"
server_key_version: "encryptionkey@0"
I20260807 02:55:31.695215 2659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "a8d4c1899a2947de86b5fa41021e6d3e"
format_stamp: "Formatted at 2026-08-07 02:55:31 on dist-test-slave-3h5h"
server_key: "64f614b5d07a3104c5ef64db7ddcdf67"
server_key_iv: "9b77e8441816ac2f53e3d5507eeb02ab"
server_key_version: "encryptionkey@0"
I20260807 02:55:31.701524 2659 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 02:55:31.705468 2687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:31.706947 2659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:55:31.707161 2659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "a8d4c1899a2947de86b5fa41021e6d3e"
format_stamp: "Formatted at 2026-08-07 02:55:31 on dist-test-slave-3h5h"
server_key: "64f614b5d07a3104c5ef64db7ddcdf67"
server_key_iv: "9b77e8441816ac2f53e3d5507eeb02ab"
server_key_version: "encryptionkey@0"
I20260807 02:55:31.707391 2659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:31.738933 2659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:31.743712 2659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:31.744063 2659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:31.745190 2659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:31.746685 2659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:31.746767 2659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:31.746953 2659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:31.747016 2659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:31.784053 2659 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:37139
I20260807 02:55:31.784173 2853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:37139 every 8 connection(s)
I20260807 02:55:31.785490 2659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:55:31.789168 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 2659
I20260807 02:55:31.789369 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:55:31.789975 1509 external_mini_cluster.cc:1428] Setting key 4edc3e9ffa501b2eefc54ef157f6f54d
Aug 07 02:55:31 dist-test-slave-3h5h krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:31.808794 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:31.788560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:57149 (local address 127.1.121.126:35425)
0807 02:55:31.789170 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:31.789178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:31.790379 (+ 1201us) server_negotiation.cc:407] Connection header received
0807 02:55:31.792612 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:31.792616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:31.792682 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:31.792780 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:31.795017 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:31.795781 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:31.797472 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:31.797969 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:31.798122 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:31.802402 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:31.802407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:31.802411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:31.802447 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:31.805552 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:31.806379 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:31.806384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:31.806386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:31.806518 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:31.806963 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:31.806973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:31.806975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:31.807250 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:31.807409 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:31.808162 (+ 753us) server_negotiation.cc:299] Negotiation successful
0807 02:55:31.808404 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":305,"threads_started":1}
I20260807 02:55:31.809882 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:31.789189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:35425 (local address 127.1.121.66:57149)
0807 02:55:31.790181 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:31.790238 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:31.792302 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:31.793089 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:31.793101 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:31.793933 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:31.794795 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:31.794823 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:31.795955 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:31.795960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:31.797316 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:31.797330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:31.798212 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:31.798221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:31.798458 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:31.799240 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:31.799279 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:31.802150 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:31.805709 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:31.805719 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:31.805738 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:31.806235 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:31.806633 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:31.806638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:31.806642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:31.806838 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:31.807384 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:31.807392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:31.807753 (+ 361us) client_negotiation.cc:769] Sending connection context
0807 02:55:31.808027 (+ 274us) client_negotiation.cc:240] Negotiation successful
0807 02:55:31.808591 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":384,"threads_started":1}
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
I20260807 02:55:31.811800 2854 heartbeater.cc:347] Connected to a master server at 127.1.121.126:35425
I20260807 02:55:31.812139 2854 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:31.813024 2854 heartbeater.cc:511] Master 127.1.121.126:35425 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
I20260807 02:55:31.814702 2381 ts_manager.cc:194] Registered new tserver with Master: a8d4c1899a2947de86b5fa41021e6d3e (127.1.121.66:37139)
I20260807 02:55:31.815608 2381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:57149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:55:31.841763 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35425
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:32.193285 2863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:32.193639 2863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:32.193827 2863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:32.201488 2863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:32.201650 2863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:32.201821 2863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:55:32.212702 2863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:32.214003 2863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:32.215358 2863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:32.229437 2868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:32.229413 2871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:32.229367 2869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:32.230878 2863 server_base.cc:1034] running on GCE node
I20260807 02:55:32.231702 2863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:32.232717 2863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:32.233953 2863 hybrid_clock.cc:648] HybridClock initialized: now 1786071332233900 us; error 59 us; skew 500 ppm
I20260807 02:55:32.234306 2863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:32 dist-test-slave-3h5h krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:32.237766 2863 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:55:32.239267 2863 webserver.cc:460] Webserver started at http://127.1.121.67:36705/ using document root <none> and password file <none>
I20260807 02:55:32.239961 2863 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:32.240093 2863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:32.240409 2863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:32.243084 2648 heartbeater.cc:503] Master 127.1.121.126:35425 was elected leader, sending a full tablet report...
I20260807 02:55:32.243731 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "0283c67d309c4c83a9bc1073c47bb429"
format_stamp: "Formatted at 2026-08-07 02:55:32 on dist-test-slave-3h5h"
server_key: "f10bb8df2e8a3f92654d7d00ad31dbc8"
server_key_iv: "e61980c5d7f4119caa423abf65662a32"
server_key_version: "encryptionkey@0"
I20260807 02:55:32.244514 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "0283c67d309c4c83a9bc1073c47bb429"
format_stamp: "Formatted at 2026-08-07 02:55:32 on dist-test-slave-3h5h"
server_key: "f10bb8df2e8a3f92654d7d00ad31dbc8"
server_key_iv: "e61980c5d7f4119caa423abf65662a32"
server_key_version: "encryptionkey@0"
I20260807 02:55:32.250089 2863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 02:55:32.254045 2879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:32.255467 2863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:55:32.255659 2863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "0283c67d309c4c83a9bc1073c47bb429"
format_stamp: "Formatted at 2026-08-07 02:55:32 on dist-test-slave-3h5h"
server_key: "f10bb8df2e8a3f92654d7d00ad31dbc8"
server_key_iv: "e61980c5d7f4119caa423abf65662a32"
server_key_version: "encryptionkey@0"
I20260807 02:55:32.255856 2863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:32.272209 2863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:32.276754 2863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:32.277135 2863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:32.278234 2863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:32.279666 2863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:32.279771 2863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:32.279995 2863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:32.280074 2863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:32.325830 2863 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:35855
I20260807 02:55:32.325858 3050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:35855 every 8 connection(s)
I20260807 02:55:32.327534 2863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:55:32.333463 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 2863
I20260807 02:55:32.333673 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:55:32.334137 1509 external_mini_cluster.cc:1428] Setting key db2192f504a015b84f67572a871bf1e2
Aug 07 02:55:32 dist-test-slave-3h5h krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:32.350482 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.330472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:36803 (local address 127.1.121.126:35425)
0807 02:55:32.330968 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.330975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.332112 (+ 1137us) server_negotiation.cc:407] Connection header received
0807 02:55:32.334265 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.334272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.334374 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.334494 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:32.336707 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.337701 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.339222 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.339624 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.339751 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.343439 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:32.343444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:32.343448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:32.343486 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:32.346802 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.347491 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.347496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.347498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.347630 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.348036 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.348040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.348042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.348368 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:32.348543 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.349594 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.349827 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260807 02:55:32.351207 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.331096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:35425 (local address 127.1.121.67:36803)
0807 02:55:32.331951 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:32.332001 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:32.333983 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:32.334864 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:32.334873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:32.335781 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:32.336506 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.336539 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.337847 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.337852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.339071 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.339083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.339856 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.339863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.340100 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.340816 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:32.340880 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:32.343211 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:32.346933 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:32.346940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:32.346953 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:32.347379 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:32.347726 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:32.347729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:32.347732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:32.347916 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:32.348536 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:32.348546 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:32.349007 (+ 461us) client_negotiation.cc:769] Sending connection context
0807 02:55:32.349380 (+ 373us) client_negotiation.cc:240] Negotiation successful
0807 02:55:32.349887 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":353,"threads_started":1}
I20260807 02:55:32.352746 3052 heartbeater.cc:347] Connected to a master server at 127.1.121.126:35425
I20260807 02:55:32.353080 3052 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:32.353863 3052 heartbeater.cc:511] Master 127.1.121.126:35425 requested a full tablet report, sending...
I20260807 02:55:32.355229 2381 ts_manager.cc:194] Registered new tserver with Master: 0283c67d309c4c83a9bc1073c47bb429 (127.1.121.67:35855)
I20260807 02:55:32.356065 2381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:36803
I20260807 02:55:32.363126 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:55:32.386174 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.371365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.1.121.126:35425)
0807 02:55:32.371556 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.371561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.372301 (+ 740us) server_negotiation.cc:407] Connection header received
0807 02:55:32.372448 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.372452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.372557 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.372655 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:32.374813 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.375793 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.377361 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.377773 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.377876 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.380380 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:32.380385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:32.380393 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:32.380434 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:32.383465 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.384091 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.384096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.384098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.384236 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.384709 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.384714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.384716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.385174 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:32.385381 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.385652 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.385847 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:55:32.394972 2381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:55:32.403939 2381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:55:32.441202 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.421122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.1.121.66:37139)
0807 02:55:32.432640 (+ 11518us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.432652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.432690 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:55:32.432801 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.432811 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.433097 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.433314 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:32.434095 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.435502 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.438297 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.439900 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.440065 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.440330 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.440658 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.440871 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11335,"thread_start_us":11228,"threads_started":1}
I20260807 02:55:32.441452 3075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.420761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:37139 (local address 127.0.0.1:41620)
0807 02:55:32.430546 (+ 9785us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:32.430577 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:32.430736 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:32.433333 (+ 2597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:32.433337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:32.433367 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:32.433873 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.433888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.435641 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.435647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.438127 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.438140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.440142 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.440150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.440363 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.440429 (+ 66us) client_negotiation.cc:769] Sending connection context
0807 02:55:32.440610 (+ 181us) client_negotiation.cc:240] Negotiation successful
0807 02:55:32.440865 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9621,"mutex_wait_us":265,"thread_start_us":9292,"threads_started":1}
I20260807 02:55:32.441569 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.420021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:35955 (local address 127.0.0.1:56298)
0807 02:55:32.421152 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:32.421212 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:32.421480 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:32.432238 (+ 10758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:32.432250 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:32.432294 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:32.432987 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.433000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.435253 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.435260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.438366 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.438383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.440522 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.440531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.440740 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.440774 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 02:55:32.440899 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 02:55:32.441034 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":38,"thread_start_us":539,"threads_started":1}
I20260807 02:55:32.442773 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.420274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.1.121.65:35955)
0807 02:55:32.430982 (+ 10708us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.430994 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.431028 (+ 34us) server_negotiation.cc:407] Connection header received
0807 02:55:32.431141 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.431149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.431599 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.431868 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:32.433199 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.435046 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.438539 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.440127 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.440275 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.440503 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.441720 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.442349 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10491,"thread_start_us":10363,"threads_started":1}
I20260807 02:55:32.442921 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.422000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.1.121.67:35855)
0807 02:55:32.431455 (+ 9455us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.431465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.433221 (+ 1756us) server_negotiation.cc:407] Connection header received
0807 02:55:32.433539 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.433551 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.433749 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.434047 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:32.434787 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.436609 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.439258 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.440582 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.440711 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.440972 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.442361 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.442536 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9224,"thread_start_us":9823,"threads_started":1}
I20260807 02:55:32.443349 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.421624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:35855 (local address 127.0.0.1:51006)
0807 02:55:32.433072 (+ 11448us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:32.433099 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:32.433243 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:32.434002 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:32.434007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:32.434052 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:32.434549 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.434561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.436872 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.436883 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.439078 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.439091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.441229 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.441236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.441480 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.441531 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 02:55:32.441646 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 02:55:32.441798 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11307,"mutex_wait_us":2,"thread_start_us":11076,"threads_started":1}
I20260807 02:55:32.446434 2756 tablet_service.cc:1467] Processing CreateTablet for tablet 869b90f5048e4cd38d5abfdaa81ce2b0 (DEFAULT_TABLE table=test-table [id=d4b55d85949242c0b3e3b45e7a5b8186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:32.446504 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 869b90f5048e4cd38d5abfdaa81ce2b0 (DEFAULT_TABLE table=test-table [id=d4b55d85949242c0b3e3b45e7a5b8186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:32.447353 2953 tablet_service.cc:1467] Processing CreateTablet for tablet 869b90f5048e4cd38d5abfdaa81ce2b0 (DEFAULT_TABLE table=test-table [id=d4b55d85949242c0b3e3b45e7a5b8186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:32.448078 2756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869b90f5048e4cd38d5abfdaa81ce2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:32.448180 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869b90f5048e4cd38d5abfdaa81ce2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:32.448796 2953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869b90f5048e4cd38d5abfdaa81ce2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:32.465643 3082 tablet_bootstrap.cc:492] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4: Bootstrap starting.
I20260807 02:55:32.467067 3080 tablet_bootstrap.cc:492] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e: Bootstrap starting.
I20260807 02:55:32.467955 3084 tablet_bootstrap.cc:492] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429: Bootstrap starting.
I20260807 02:55:32.471266 3084 tablet_bootstrap.cc:654] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:32.471542 3082 tablet_bootstrap.cc:654] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:32.471662 3080 tablet_bootstrap.cc:654] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:32.472453 3084 log.cc:824] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:32.473052 3082 log.cc:824] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:32.473078 3080 log.cc:824] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:32.475878 3084 tablet_bootstrap.cc:492] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429: No bootstrap required, opened a new log
I20260807 02:55:32.476176 3084 ts_tablet_manager.cc:1397] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260807 02:55:32.480111 3082 tablet_bootstrap.cc:492] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4: No bootstrap required, opened a new log
I20260807 02:55:32.480227 3080 tablet_bootstrap.cc:492] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e: No bootstrap required, opened a new log
I20260807 02:55:32.480484 3080 ts_tablet_manager.cc:1397] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260807 02:55:32.480451 3082 ts_tablet_manager.cc:1397] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260807 02:55:32.485070 3084 raft_consensus.cc:348] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } }
I20260807 02:55:32.485631 3084 raft_consensus.cc:374] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:32.485767 3084 raft_consensus.cc:729] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0283c67d309c4c83a9bc1073c47bb429, State: Initialized, Role: FOLLOWER
I20260807 02:55:32.486720 3084 consensus_queue.cc:260] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } }
I20260807 02:55:32.489554 3052 heartbeater.cc:503] Master 127.1.121.126:35425 was elected leader, sending a full tablet report...
I20260807 02:55:32.490260 3084 ts_tablet_manager.cc:1428] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260807 02:55:32.493052 3082 raft_consensus.cc:348] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } }
I20260807 02:55:32.493168 3080 raft_consensus.cc:348] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } }
I20260807 02:55:32.493520 3082 raft_consensus.cc:374] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:32.493641 3082 raft_consensus.cc:729] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7366fa4c67946c394f12d35858460f4, State: Initialized, Role: FOLLOWER
I20260807 02:55:32.493670 3080 raft_consensus.cc:374] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:32.493798 3080 raft_consensus.cc:729] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d4c1899a2947de86b5fa41021e6d3e, State: Initialized, Role: FOLLOWER
I20260807 02:55:32.494386 3082 consensus_queue.cc:260] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } }
I20260807 02:55:32.494540 3080 consensus_queue.cc:260] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } }
I20260807 02:55:32.496402 3082 ts_tablet_manager.cc:1428] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260807 02:55:32.496681 2854 heartbeater.cc:503] Master 127.1.121.126:35425 was elected leader, sending a full tablet report...
I20260807 02:55:32.497344 3080 ts_tablet_manager.cc:1428] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260807 02:55:32.537858 2856 tablet.cc:2369] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:55:32.579788 3053 tablet.cc:2369] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:55:32.713003 2650 tablet.cc:2369] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:55:32.733248 3092 raft_consensus.cc:482] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:32.733606 3092 raft_consensus.cc:504] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } }
I20260807 02:55:32.736246 3092 leader_election.cc:288] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0283c67d309c4c83a9bc1073c47bb429 (127.1.121.67:35855), a8d4c1899a2947de86b5fa41021e6d3e (127.1.121.66:37139)
I20260807 02:55:32.744648 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.737147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:43963 (local address 127.1.121.66:37139)
0807 02:55:32.737432 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.737439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.738108 (+ 669us) server_negotiation.cc:407] Connection header received
0807 02:55:32.738304 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.738309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.738386 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.738487 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:32.739351 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.740024 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.742730 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.743562 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.743743 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.743836 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.744210 (+ 374us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.744361 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260807 02:55:32.744692 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.736590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:55875 (local address 127.1.121.67:35855)
0807 02:55:32.736884 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.736891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.737936 (+ 1045us) server_negotiation.cc:407] Connection header received
0807 02:55:32.738093 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.738100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.738188 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.738281 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:32.739049 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.740093 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.742692 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.743570 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.743800 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.743880 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.744242 (+ 362us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.744382 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 02:55:32.745088 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.736671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:35855 (local address 127.1.121.65:55875)
0807 02:55:32.737817 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:32.737833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:32.737963 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:32.738323 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:32.738328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:32.738352 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:32.738866 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.738881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.740238 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.740241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.742515 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.742528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.743880 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.743891 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.744017 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.744067 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 02:55:32.744206 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 02:55:32.744361 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":87,"thread_start_us":631,"threads_started":1}
I20260807 02:55:32.745170 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.736905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:37139 (local address 127.1.121.65:43963)
0807 02:55:32.737981 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:32.737999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:32.738149 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:32.738528 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:32.738532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:32.738558 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:32.739117 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.739134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.740211 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.740216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.742536 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:32.742551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.743823 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.743831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:32.744012 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.744061 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 02:55:32.744185 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 02:55:32.744358 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":4,"thread_start_us":667,"threads_started":1}
I20260807 02:55:32.746240 2987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869b90f5048e4cd38d5abfdaa81ce2b0" candidate_uuid: "b7366fa4c67946c394f12d35858460f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0283c67d309c4c83a9bc1073c47bb429" is_pre_election: true
I20260807 02:55:32.746383 2790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869b90f5048e4cd38d5abfdaa81ce2b0" candidate_uuid: "b7366fa4c67946c394f12d35858460f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" is_pre_election: true
I20260807 02:55:32.746987 2987 raft_consensus.cc:2454] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7366fa4c67946c394f12d35858460f4 in term 0.
I20260807 02:55:32.747081 2790 raft_consensus.cc:2454] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7366fa4c67946c394f12d35858460f4 in term 0.
I20260807 02:55:32.749087 2483 leader_election.cc:302] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0283c67d309c4c83a9bc1073c47bb429, b7366fa4c67946c394f12d35858460f4; no voters:
I20260807 02:55:32.749524 3092 raft_consensus.cc:2781] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:55:32.749702 3092 raft_consensus.cc:482] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:55:32.749820 3092 raft_consensus.cc:3037] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:32.752553 3092 raft_consensus.cc:504] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } }
I20260807 02:55:32.753479 3092 leader_election.cc:288] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [CANDIDATE]: Term 1 election: Requested vote from peers 0283c67d309c4c83a9bc1073c47bb429 (127.1.121.67:35855), a8d4c1899a2947de86b5fa41021e6d3e (127.1.121.66:37139)
I20260807 02:55:32.753842 2987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869b90f5048e4cd38d5abfdaa81ce2b0" candidate_uuid: "b7366fa4c67946c394f12d35858460f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0283c67d309c4c83a9bc1073c47bb429"
I20260807 02:55:32.754022 2987 raft_consensus.cc:3037] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:32.754159 2790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869b90f5048e4cd38d5abfdaa81ce2b0" candidate_uuid: "b7366fa4c67946c394f12d35858460f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d4c1899a2947de86b5fa41021e6d3e"
I20260807 02:55:32.754406 2790 raft_consensus.cc:3037] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:32.756374 2987 raft_consensus.cc:2454] T 869b90f5048e4cd38d5abfdaa81ce2b0 P 0283c67d309c4c83a9bc1073c47bb429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7366fa4c67946c394f12d35858460f4 in term 1.
I20260807 02:55:32.756768 2483 leader_election.cc:302] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0283c67d309c4c83a9bc1073c47bb429, b7366fa4c67946c394f12d35858460f4; no voters:
I20260807 02:55:32.757038 2790 raft_consensus.cc:2454] T 869b90f5048e4cd38d5abfdaa81ce2b0 P a8d4c1899a2947de86b5fa41021e6d3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7366fa4c67946c394f12d35858460f4 in term 1.
I20260807 02:55:32.757122 3092 raft_consensus.cc:2781] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:32.757752 3092 raft_consensus.cc:686] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [term 1 LEADER]: Becoming Leader. State: Replica: b7366fa4c67946c394f12d35858460f4, State: Running, Role: LEADER
I20260807 02:55:32.758459 3092 consensus_queue.cc:237] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } }
I20260807 02:55:32.763971 2374 catalog_manager.cc:5495] T 869b90f5048e4cd38d5abfdaa81ce2b0 P b7366fa4c67946c394f12d35858460f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7366fa4c67946c394f12d35858460f4 (127.1.121.65). New cstate: current_term: 1 leader_uuid: "b7366fa4c67946c394f12d35858460f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7366fa4c67946c394f12d35858460f4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0283c67d309c4c83a9bc1073c47bb429" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8d4c1899a2947de86b5fa41021e6d3e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37139 } health_report { overall_health: UNKNOWN } } }
Aug 07 02:55:32 dist-test-slave-3h5h krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:55:32.810559 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.793167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.1.121.65:35955)
0807 02:55:32.793416 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.793423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.794203 (+ 780us) server_negotiation.cc:407] Connection header received
0807 02:55:32.794397 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.794401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.794517 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.794613 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:32.796560 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.797450 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.798791 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.799308 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.799418 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.803714 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:32.803732 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:32.803750 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:32.803798 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:32.807598 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.808345 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.808354 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.808357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.808542 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.808945 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.808949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.808951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.809334 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:32.809597 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.809856 (+ 259us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.810208 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Aug 07 02:55:32 dist-test-slave-3h5h krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:55:32 dist-test-slave-3h5h krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:55:32.875751 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.859440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.1.121.65:35955)
0807 02:55:32.859683 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.859691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.860122 (+ 431us) server_negotiation.cc:407] Connection header received
0807 02:55:32.860374 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.860379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.860459 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.860578 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:32.862618 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.863447 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.864791 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.865369 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.865548 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.869853 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:32.869859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:32.869862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:32.869900 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:32.872711 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.873487 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.873496 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.873502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.873655 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.874106 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.874112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.874118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.874410 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:32.874795 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.875113 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.875383 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Aug 07 02:55:32 dist-test-slave-3h5h krb5kdc[2303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:55:32 dist-test-slave-3h5h krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:55:32.934665 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.916816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.1.121.65:35955)
0807 02:55:32.917128 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.917138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.917692 (+ 554us) server_negotiation.cc:407] Connection header received
0807 02:55:32.917976 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.917982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.918082 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.918216 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:32.920103 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.921149 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.922640 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.923156 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.923320 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.928131 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:32.928136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:32.928140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:32.928180 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:32.931218 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.932165 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.932172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.932185 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.932364 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.932823 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.932853 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.932859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.933215 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:32.933568 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.933961 (+ 393us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.934230 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260807 02:55:32.937651 2518 tablet_service.cc:2908] Rejecting scan request for tablet 869b90f5048e4cd38d5abfdaa81ce2b0: Uninitialized: clean time has not yet been initialized
I20260807 02:55:32.941866 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 2465
I20260807 02:55:32.955446 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:32.942944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.1.121.65:35955)
0807 02:55:32.943181 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:32.943190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:32.943486 (+ 296us) server_negotiation.cc:407] Connection header received
0807 02:55:32.943656 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:32.943660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:32.943719 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:32.943797 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:32.945181 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.945860 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.947202 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:32.947580 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:32.947677 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:32.950240 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:32.950245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:32.950249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:32.950297 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:32.952955 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.953660 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.953664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.953666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.953801 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:32.954176 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:32.954179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:32.954181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:32.954436 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:32.954614 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:32.954910 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 02:55:32.955097 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:55:32.956218 2642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:33.055266 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 2465
I20260807 02:55:33.085073 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 2659
Aug 07 02:55:33 dist-test-slave-3h5h krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:55:33.100802 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:33.085942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.1.121.66:37139)
0807 02:55:33.086156 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:33.086163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:33.086257 (+ 94us) server_negotiation.cc:407] Connection header received
0807 02:55:33.086421 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:33.086425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:33.086523 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:33.086620 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:33.087839 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:33.088786 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:33.090138 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:33.090604 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:33.090717 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:33.094825 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:33.094841 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:33.094859 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:33.094904 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:33.098049 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:33.098701 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:33.098710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:33.098713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:33.098871 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:33.099233 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:33.099238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:33.099240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:33.099702 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:33.099951 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:33.100218 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 02:55:33.100384 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 02:55:33.101804 2846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:33.192668 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 2659
I20260807 02:55:33.219887 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 2863
Aug 07 02:55:33 dist-test-slave-3h5h krb5kdc[2303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:55:33.235039 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:33.220755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.1.121.67:35855)
0807 02:55:33.221034 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:33.221039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:33.221059 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:55:33.221209 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:33.221213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:33.221278 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:33.221375 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:33.222567 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:33.223281 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:33.224585 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:33.225025 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:33.225119 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:33.229531 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:33.229546 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:33.229565 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:33.229599 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:33.232507 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:33.233312 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:33.233320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:33.233322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:33.233448 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:33.233793 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:33.233797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:33.233799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:33.234144 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:33.234363 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:33.234594 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 02:55:33.234762 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:55:33.235697 3045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:33.328663 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 2863
I20260807 02:55:33.352495 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 2327
I20260807 02:55:33.353965 2423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:33.464385 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 2327
2026-08-07T02:55:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4842 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:55:33 dist-test-slave-3h5h krb5kdc[3128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:55:33 dist-test-slave-3h5h krb5kdc[3128](info): set up 2 sockets
Aug 07 02:55:33 dist-test-slave-3h5h krb5kdc[3128](info): commencing operation
krb5kdc: starting...
W20260807 02:55:34.831342 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:55:34 dist-test-slave-3h5h krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:55:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:55:35.061661 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41433
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:41433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:35.416747 3147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:35.417160 3147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:35.417317 3147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:35.424690 3147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:55:35.424835 3147 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 02:55:35.424964 3147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:35.425076 3147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:35.425169 3147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:55:35.425278 3147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:55:35.435684 3147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:41433
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:35.437141 3147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:35.438570 3147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:35.451076 3153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:35.451122 3154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:35.451594 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:35.452454 3147 server_base.cc:1034] running on GCE node
I20260807 02:55:35.453351 3147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:35.454789 3147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:35.456080 3147 hybrid_clock.cc:648] HybridClock initialized: now 1786071335456014 us; error 35 us; skew 500 ppm
I20260807 02:55:35.456435 3147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:35 dist-test-slave-3h5h krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:35.459872 3147 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:55:35.461544 3147 webserver.cc:460] Webserver started at http://127.1.121.126:34743/ using document root <none> and password file <none>
I20260807 02:55:35.462281 3147 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:35.462414 3147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:35.462765 3147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:35.465827 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "456f3fe7048e4d558476f6f75e814fd5"
format_stamp: "Formatted at 2026-08-07 02:55:35 on dist-test-slave-3h5h"
server_key: "f8243c2a91c189c459c117dd99bc817a"
server_key_iv: "768d1bff75ae8ca23e00482698c85cba"
server_key_version: "encryptionkey@0"
I20260807 02:55:35.466637 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "456f3fe7048e4d558476f6f75e814fd5"
format_stamp: "Formatted at 2026-08-07 02:55:35 on dist-test-slave-3h5h"
server_key: "f8243c2a91c189c459c117dd99bc817a"
server_key_iv: "768d1bff75ae8ca23e00482698c85cba"
server_key_version: "encryptionkey@0"
I20260807 02:55:35.471776 3147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 02:55:35.475543 3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:35.477137 3147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:55:35.477335 3147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "456f3fe7048e4d558476f6f75e814fd5"
format_stamp: "Formatted at 2026-08-07 02:55:35 on dist-test-slave-3h5h"
server_key: "f8243c2a91c189c459c117dd99bc817a"
server_key_iv: "768d1bff75ae8ca23e00482698c85cba"
server_key_version: "encryptionkey@0"
I20260807 02:55:35.477562 3147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:35.491447 3147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:35.496166 3147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:35.496482 3147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:35.520113 3147 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:41433
I20260807 02:55:35.520134 3251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:41433 every 8 connection(s)
I20260807 02:55:35.521653 3147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:55:35.522974 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 3147
I20260807 02:55:35.523216 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:55:35.524025 1509 external_mini_cluster.cc:1428] Setting key d20e1600bbeba3ee73eb3df7b396ab50
I20260807 02:55:35.526664 3255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:35.538887 3255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5: Bootstrap starting.
I20260807 02:55:35.542043 3255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:35.543342 3255 log.cc:824] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:55:35 dist-test-slave-3h5h krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:35.546850 3255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5: No bootstrap required, opened a new log
I20260807 02:55:35.555104 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:35.526263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.1.121.126:41433)
0807 02:55:35.527069 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:35.527081 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:35.527122 (+ 41us) server_negotiation.cc:407] Connection header received
0807 02:55:35.528727 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:35.528748 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:35.529222 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:35.529560 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:35.531422 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:35.533128 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:35.534835 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:35.535639 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:35.535789 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:35.546870 (+ 11081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:35.546879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:35.546894 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:35.546931 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:35.550617 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:35.551561 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:35.551573 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:35.551584 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:35.551801 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:35.552236 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:35.552239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:35.552241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:35.552777 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:35.553164 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:35.553509 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 02:55:35.553922 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":393,"threads_started":1}
I20260807 02:55:35.556219 3255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456f3fe7048e4d558476f6f75e814fd5" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41433 } }
I20260807 02:55:35.556592 3255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:35.556731 3255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456f3fe7048e4d558476f6f75e814fd5, State: Initialized, Role: FOLLOWER
I20260807 02:55:35.557308 3255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456f3fe7048e4d558476f6f75e814fd5" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41433 } }
I20260807 02:55:35.557586 3255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:55:35.557746 3255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:55:35.557942 3255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:35.560149 3255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456f3fe7048e4d558476f6f75e814fd5" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41433 } }
I20260807 02:55:35.560690 3255 leader_election.cc:302] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 456f3fe7048e4d558476f6f75e814fd5; no voters:
I20260807 02:55:35.561292 3255 leader_election.cc:288] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:55:35.561451 3261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:35.562901 3261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [term 1 LEADER]: Becoming Leader. State: Replica: 456f3fe7048e4d558476f6f75e814fd5, State: Running, Role: LEADER
I20260807 02:55:35.563481 3255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:55:35.563447 3261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456f3fe7048e4d558476f6f75e814fd5" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41433 } }
I20260807 02:55:35.566501 3263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 456f3fe7048e4d558476f6f75e814fd5. Latest consensus state: current_term: 1 leader_uuid: "456f3fe7048e4d558476f6f75e814fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456f3fe7048e4d558476f6f75e814fd5" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41433 } } }
I20260807 02:55:35.566921 3263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:35.567523 3262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "456f3fe7048e4d558476f6f75e814fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456f3fe7048e4d558476f6f75e814fd5" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41433 } } }
I20260807 02:55:35.567888 3262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:35.570034 3272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:55:35.576162 3272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:55:35.586285 3272 catalog_manager.cc:1347] Generated new cluster ID: a3abf2786c394d03a575aa99ced7a577
I20260807 02:55:35.586478 3272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:55:35.612468 3272 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:55:35.614092 3272 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:55:35.622069 3272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5: Generated new TSK 0
I20260807 02:55:35.623020 3272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:55:35.692536 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41433
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:36.044471 3291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:36.044785 3291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:36.044956 3291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:36.052623 3291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:36.052742 3291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:36.052881 3291 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:55:36.053049 3291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:55:36.063530 3291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.121.126:41433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:36.065076 3291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:36.066515 3291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:36.080821 3304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:36.080953 3299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:36.080955 3300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:36.081765 3291 server_base.cc:1034] running on GCE node
I20260807 02:55:36.082448 3291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:36.083416 3291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:36.084681 3291 hybrid_clock.cc:648] HybridClock initialized: now 1786071336084607 us; error 42 us; skew 500 ppm
I20260807 02:55:36.085189 3291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:36 dist-test-slave-3h5h krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:36.088506 3291 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:55:36.090103 3291 webserver.cc:460] Webserver started at http://127.1.121.65:36037/ using document root <none> and password file <none>
I20260807 02:55:36.090824 3291 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:36.090960 3291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:36.091308 3291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:36.094556 3291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "39e816245f36447ebceb5b4ddf61675a"
format_stamp: "Formatted at 2026-08-07 02:55:36 on dist-test-slave-3h5h"
server_key: "e0046c3da6f4c95ae32aea9c498fe07c"
server_key_iv: "9524e629026d636bce85379bc6c5fd1c"
server_key_version: "encryptionkey@0"
I20260807 02:55:36.095319 3291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "39e816245f36447ebceb5b4ddf61675a"
format_stamp: "Formatted at 2026-08-07 02:55:36 on dist-test-slave-3h5h"
server_key: "e0046c3da6f4c95ae32aea9c498fe07c"
server_key_iv: "9524e629026d636bce85379bc6c5fd1c"
server_key_version: "encryptionkey@0"
I20260807 02:55:36.100574 3291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 02:55:36.104112 3313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:36.105609 3291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:55:36.105794 3291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "39e816245f36447ebceb5b4ddf61675a"
format_stamp: "Formatted at 2026-08-07 02:55:36 on dist-test-slave-3h5h"
server_key: "e0046c3da6f4c95ae32aea9c498fe07c"
server_key_iv: "9524e629026d636bce85379bc6c5fd1c"
server_key_version: "encryptionkey@0"
I20260807 02:55:36.106025 3291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:36.137799 3291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:36.142055 3291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:36.142371 3291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:36.145439 3291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:36.145569 3291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:36.145753 3291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:36.145838 3291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:55:36 dist-test-slave-3h5h krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:36.172477 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:36.146266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:50825 (local address 127.1.121.126:41433)
0807 02:55:36.146879 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:36.146885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:36.148243 (+ 1358us) server_negotiation.cc:407] Connection header received
0807 02:55:36.151546 (+ 3303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:36.151553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:36.151668 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:36.151818 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:36.154433 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.155544 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.157926 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.158739 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.158904 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:36.164807 (+ 5903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:36.164816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:36.164821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:36.164902 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:36.168380 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:36.169722 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:36.169729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:36.169732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:36.169919 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:36.170466 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:36.170471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:36.170474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:36.170822 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:36.170984 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:36.171848 (+ 864us) server_negotiation.cc:299] Negotiation successful
0807 02:55:36.172079 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":355,"threads_started":1}
I20260807 02:55:36.174034 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:36.147025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41433 (local address 127.1.121.65:50825)
0807 02:55:36.148015 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:36.148080 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:36.151051 (+ 2971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:36.152148 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:36.152166 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:36.153086 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:36.154215 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:36.154252 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.155728 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.155735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:36.157748 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:36.157768 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.159041 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.159052 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:36.159366 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:36.160280 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:36.160305 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:36.164479 (+ 4174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:36.168599 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:36.168610 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:36.168640 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:36.169552 (+ 912us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:36.170045 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:36.170050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:36.170054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:36.170341 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:36.170980 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:36.170988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:36.171453 (+ 465us) client_negotiation.cc:769] Sending connection context
0807 02:55:36.171733 (+ 280us) client_negotiation.cc:240] Negotiation successful
0807 02:55:36.172301 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":372,"threads_started":1}
I20260807 02:55:36.196965 3291 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:39735
I20260807 02:55:36.196995 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:39735 every 8 connection(s)
I20260807 02:55:36.198233 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:36.181622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41433 (local address 127.1.121.65:60669)
0807 02:55:36.182428 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:36.182451 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:36.182652 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:36.183178 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:36.183187 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:36.184092 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:36.184814 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:36.184851 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.186415 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.186419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:36.187970 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:36.187991 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.189217 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.189222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:36.189347 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:36.190097 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:36.190130 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:36.190954 (+ 824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:36.195193 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:36.195197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:36.195200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:36.195847 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:36.196311 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:36.196318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:36.196324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:36.196514 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:36.197233 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:36.197242 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:36.197453 (+ 211us) client_negotiation.cc:769] Sending connection context
0807 02:55:36.197640 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 02:55:36.197899 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":418,"threads_started":1}
I20260807 02:55:36.198411 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:36.181498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:60669 (local address 127.1.121.126:41433)
0807 02:55:36.181672 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:36.181677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:36.182565 (+ 888us) server_negotiation.cc:407] Connection header received
0807 02:55:36.182813 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:36.182823 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:36.182923 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:36.183038 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:36.185017 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.186273 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.188173 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.189018 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.189163 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:36.191188 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:36.191195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:36.191199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:36.191237 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:36.195040 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:36.195995 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:36.196004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:36.196011 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:36.196179 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:36.196676 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:36.196684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:36.196689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:36.197110 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:36.197263 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:36.197775 (+ 512us) server_negotiation.cc:299] Negotiation successful
0807 02:55:36.198044 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 02:55:36.198665 3291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:55:36.201267 3479 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41433
I20260807 02:55:36.201505 3479 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:36.202191 3479 heartbeater.cc:511] Master 127.1.121.126:41433 requested a full tablet report, sending...
I20260807 02:55:36.204100 3188 ts_manager.cc:194] Registered new tserver with Master: 39e816245f36447ebceb5b4ddf61675a (127.1.121.65:39735)
I20260807 02:55:36.205392 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 3291
I20260807 02:55:36.205624 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:55:36.205842 3188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:50825
I20260807 02:55:36.206326 1509 external_mini_cluster.cc:1428] Setting key ca2e46178cdee370c900c0b663a5ca56
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:55:36.256739 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41433
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:36.599109 3485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:36.599459 3485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:36.599650 3485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:36.607787 3485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:36.607950 3485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:36.608114 3485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:55:36.608258 3485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:55:36.618785 3485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.121.126:41433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:36.620180 3485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:36.621620 3485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:36.635284 3495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:36.635344 3493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:36.635396 3492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:36.638448 3485 server_base.cc:1034] running on GCE node
I20260807 02:55:36.639132 3485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:36.640168 3485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:36.641422 3485 hybrid_clock.cc:648] HybridClock initialized: now 1786071336641351 us; error 50 us; skew 500 ppm
I20260807 02:55:36.641731 3485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:36 dist-test-slave-3h5h krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:36.645304 3485 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:55:36.646952 3485 webserver.cc:460] Webserver started at http://127.1.121.66:44857/ using document root <none> and password file <none>
I20260807 02:55:36.647819 3485 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:36.647940 3485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:36.648257 3485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:36.651211 3485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "3218d94b5b434de18c2bd66536df1390"
format_stamp: "Formatted at 2026-08-07 02:55:36 on dist-test-slave-3h5h"
server_key: "f1b4e4311ca389e5c3a5f80f35dc757e"
server_key_iv: "a3ba80fc8e208ffd81e4b75ae5e2a9e9"
server_key_version: "encryptionkey@0"
I20260807 02:55:36.651963 3485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "3218d94b5b434de18c2bd66536df1390"
format_stamp: "Formatted at 2026-08-07 02:55:36 on dist-test-slave-3h5h"
server_key: "f1b4e4311ca389e5c3a5f80f35dc757e"
server_key_iv: "a3ba80fc8e208ffd81e4b75ae5e2a9e9"
server_key_version: "encryptionkey@0"
I20260807 02:55:36.657939 3485 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 02:55:36.661608 3503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:36.663180 3485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:55:36.663352 3485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "3218d94b5b434de18c2bd66536df1390"
format_stamp: "Formatted at 2026-08-07 02:55:36 on dist-test-slave-3h5h"
server_key: "f1b4e4311ca389e5c3a5f80f35dc757e"
server_key_iv: "a3ba80fc8e208ffd81e4b75ae5e2a9e9"
server_key_version: "encryptionkey@0"
I20260807 02:55:36.663589 3485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:36.712613 3485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:36.717880 3485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:36.718209 3485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:36.721428 3485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:36.721549 3485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:36.721741 3485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:36.721832 3485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:55:36 dist-test-slave-3h5h krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:36.749076 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:36.722506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:32859 (local address 127.1.121.126:41433)
0807 02:55:36.725276 (+ 2770us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:36.725283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:36.725308 (+ 25us) server_negotiation.cc:407] Connection header received
0807 02:55:36.727383 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:36.727392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:36.727520 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:36.727665 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:36.729729 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.731298 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.733377 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.734103 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.734228 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:36.740320 (+ 6092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:36.740328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:36.740336 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:36.740388 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:36.744614 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:36.745977 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:36.745986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:36.745991 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:36.746181 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:36.746838 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:36.746847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:36.746852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:36.747281 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:36.747481 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:36.748323 (+ 842us) server_negotiation.cc:299] Negotiation successful
0807 02:55:36.748585 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2635,"thread_start_us":2517,"threads_started":1}
I20260807 02:55:36.750236 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:36.723225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41433 (local address 127.1.121.66:32859)
0807 02:55:36.724523 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:36.724600 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:36.726941 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:36.727881 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:36.727890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:36.728516 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:36.729472 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:36.729514 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.731514 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.731518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:36.733133 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:36.733147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.734595 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.734613 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:36.735051 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:36.735972 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:36.736079 (+ 107us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:36.739942 (+ 3863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:36.744805 (+ 4863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:36.744823 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:36.744845 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:36.745778 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:36.746352 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:36.746360 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:36.746367 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:36.746683 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:36.747435 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:36.747446 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:36.747892 (+ 446us) client_negotiation.cc:769] Sending connection context
0807 02:55:36.748194 (+ 302us) client_negotiation.cc:240] Negotiation successful
0807 02:55:36.748804 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":541,"threads_started":1}
I20260807 02:55:36.776999 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:36.757978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41433 (local address 127.1.121.66:34247)
0807 02:55:36.758819 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:36.758846 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:36.759044 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:36.759427 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:36.759434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:36.760212 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:36.760685 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:36.760701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.762003 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.762011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:36.764092 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:36.764107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.765427 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.765437 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:36.765706 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:36.766765 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:36.766791 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:36.768547 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:36.773068 (+ 4521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:36.773073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:36.773077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:36.773713 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:36.774271 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:36.774279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:36.774285 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:36.774484 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:36.775404 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:36.775418 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:36.775717 (+ 299us) client_negotiation.cc:769] Sending connection context
0807 02:55:36.775980 (+ 263us) client_negotiation.cc:240] Negotiation successful
0807 02:55:36.776496 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":416,"threads_started":1}
I20260807 02:55:36.777755 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:36.757950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:34247 (local address 127.1.121.126:41433)
0807 02:55:36.758176 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:36.758183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:36.758949 (+ 766us) server_negotiation.cc:407] Connection header received
0807 02:55:36.759198 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:36.759202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:36.759269 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:36.759364 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:36.760944 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.761845 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.764304 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:36.765099 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:36.765343 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:36.768771 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:36.768777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:36.768780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:36.768816 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:36.772900 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:36.773870 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:36.773875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:36.773878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:36.774117 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:36.774651 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:36.774660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:36.774667 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:36.775203 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:36.775388 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:36.777114 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0807 02:55:36.777425 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 02:55:36.786607 3485 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:37645
I20260807 02:55:36.786621 3678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:37645 every 8 connection(s)
I20260807 02:55:36.787842 3485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:55:36.789965 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 3485
I20260807 02:55:36.790035 3680 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41433
I20260807 02:55:36.790203 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:55:36.790334 3680 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:36.790711 1509 external_mini_cluster.cc:1428] Setting key db9ece1b3689a3cfe98fd2251ff65f54
I20260807 02:55:36.791034 3680 heartbeater.cc:511] Master 127.1.121.126:41433 requested a full tablet report, sending...
I20260807 02:55:36.792714 3187 ts_manager.cc:194] Registered new tserver with Master: 3218d94b5b434de18c2bd66536df1390 (127.1.121.66:37645)
I20260807 02:55:36.794039 3187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:32859
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:55:36.841411 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41433
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:55:37.209059 3479 heartbeater.cc:503] Master 127.1.121.126:41433 was elected leader, sending a full tablet report...
W20260807 02:55:37.238488 3685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:37.238770 3685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:37.238870 3685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:37.246208 3685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:37.246322 3685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:37.246408 3685 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:55:37.246510 3685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:55:37.256577 3685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.121.126:41433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:37.257939 3685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:37.259223 3685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:37.273401 3692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:37.273401 3696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:37.274024 3693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:37.274158 3685 server_base.cc:1034] running on GCE node
I20260807 02:55:37.274788 3685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:37.275710 3685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:37.276958 3685 hybrid_clock.cc:648] HybridClock initialized: now 1786071337276883 us; error 33 us; skew 500 ppm
I20260807 02:55:37.277360 3685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:37 dist-test-slave-3h5h krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071337, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:37.280661 3685 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:55:37.282194 3685 webserver.cc:460] Webserver started at http://127.1.121.67:44315/ using document root <none> and password file <none>
I20260807 02:55:37.282933 3685 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:37.283023 3685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:37.283304 3685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:37.286517 3685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "fe1599b002834155b469a7796f2add9e"
format_stamp: "Formatted at 2026-08-07 02:55:37 on dist-test-slave-3h5h"
server_key: "c9f893a085bd28fe2697ee312710c877"
server_key_iv: "35a75e67c5a72acc2f444cd4d8310dc6"
server_key_version: "encryptionkey@0"
I20260807 02:55:37.287272 3685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "fe1599b002834155b469a7796f2add9e"
format_stamp: "Formatted at 2026-08-07 02:55:37 on dist-test-slave-3h5h"
server_key: "c9f893a085bd28fe2697ee312710c877"
server_key_iv: "35a75e67c5a72acc2f444cd4d8310dc6"
server_key_version: "encryptionkey@0"
I20260807 02:55:37.292608 3685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 02:55:37.296379 3707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:37.298344 3685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:55:37.298535 3685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "fe1599b002834155b469a7796f2add9e"
format_stamp: "Formatted at 2026-08-07 02:55:37 on dist-test-slave-3h5h"
server_key: "c9f893a085bd28fe2697ee312710c877"
server_key_iv: "35a75e67c5a72acc2f444cd4d8310dc6"
server_key_version: "encryptionkey@0"
I20260807 02:55:37.298840 3685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:37.320894 3685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:37.325376 3685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:37.325659 3685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:37.328769 3685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:37.328896 3685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 02:55:37.329073 3685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:37.329145 3685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:55:37 dist-test-slave-3h5h krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071337, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:37.355015 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.329826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:50165 (local address 127.1.121.126:41433)
0807 02:55:37.330370 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:37.330377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:37.331621 (+ 1244us) server_negotiation.cc:407] Connection header received
0807 02:55:37.333849 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:37.333855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:37.333957 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:37.334089 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:37.336690 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.337882 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.340227 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.341049 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.341209 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.346969 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:37.346977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:37.346984 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:37.347032 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:37.350806 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:37.351884 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:37.351893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:37.351899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:37.352122 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:37.352601 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:37.352608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:37.352614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:37.353155 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:37.353333 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:37.354059 (+ 726us) server_negotiation.cc:299] Negotiation successful
0807 02:55:37.354368 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":275,"threads_started":1}
I20260807 02:55:37.355568 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.330489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41433 (local address 127.1.121.67:50165)
0807 02:55:37.331423 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:37.331473 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:37.333517 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:37.334397 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:37.334412 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:37.335377 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:37.336461 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.336500 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.338090 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.338094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.340067 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.340080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.341327 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.341340 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.341625 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.342401 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:37.342438 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:37.346692 (+ 4254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:37.350974 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:37.350983 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:37.351007 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:37.351735 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:37.352264 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:37.352268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:37.352270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:37.352482 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:37.353294 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:37.353300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:37.353652 (+ 352us) client_negotiation.cc:769] Sending connection context
0807 02:55:37.353907 (+ 255us) client_negotiation.cc:240] Negotiation successful
0807 02:55:37.354315 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":412,"threads_started":1}
I20260807 02:55:37.380013 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.363103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:39145 (local address 127.1.121.126:41433)
0807 02:55:37.363335 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:37.363343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:37.363937 (+ 594us) server_negotiation.cc:407] Connection header received
0807 02:55:37.364089 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:37.364097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:37.364199 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:37.364332 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:37.365620 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.367019 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.368449 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.369313 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.369462 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.372582 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:37.372588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:37.372593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:37.372635 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:37.377030 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:37.377769 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:37.377775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:37.377779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:37.377934 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:37.378347 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:37.378352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:37.378356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:37.378681 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:37.378864 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:37.379318 (+ 454us) server_negotiation.cc:299] Negotiation successful
0807 02:55:37.379572 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 02:55:37.380116 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.363117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41433 (local address 127.1.121.67:39145)
0807 02:55:37.363829 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:37.363855 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:37.363970 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:37.364355 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:37.364359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:37.364921 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:37.365433 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.365451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.367175 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.367179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.368308 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.368321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.369904 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.369916 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.370140 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.370838 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:37.370871 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:37.372301 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:37.377178 (+ 4877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:37.377185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:37.377189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:37.377624 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:37.378068 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:37.378074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:37.378078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:37.378213 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:37.378817 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:37.378824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:37.379048 (+ 224us) client_negotiation.cc:769] Sending connection context
0807 02:55:37.379216 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 02:55:37.379585 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":425,"threads_started":1}
I20260807 02:55:37.387316 3685 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:35971
I20260807 02:55:37.387316 3875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:35971 every 8 connection(s)
I20260807 02:55:37.388605 3685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:55:37.390867 3877 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41433
I20260807 02:55:37.391155 3877 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:37.391885 3877 heartbeater.cc:511] Master 127.1.121.126:41433 requested a full tablet report, sending...
I20260807 02:55:37.393385 3187 ts_manager.cc:194] Registered new tserver with Master: fe1599b002834155b469a7796f2add9e (127.1.121.67:35971)
I20260807 02:55:37.394280 3187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:50165
I20260807 02:55:37.398394 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 3685
I20260807 02:55:37.398604 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:55:37.399160 1509 external_mini_cluster.cc:1428] Setting key e3d2b98aaf9702d40cbdc41b0d3ae25d
I20260807 02:55:37.403640 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:55:37 dist-test-slave-3h5h krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:55:37 dist-test-slave-3h5h krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:37.457545 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.443080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.1.121.126:41433)
0807 02:55:37.443337 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:37.443344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:37.443562 (+ 218us) server_negotiation.cc:407] Connection header received
0807 02:55:37.443827 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:37.443831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:37.443898 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:37.444011 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:37.445492 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.446258 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.447568 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.447995 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.448149 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.452046 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:37.452052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:37.452055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:37.452090 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:37.454832 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:37.455486 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:37.455491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:37.455493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:37.455624 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:37.456044 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:37.456050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:37.456053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:37.456349 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:37.456625 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:37.456961 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 02:55:37.457203 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 02:55:37.462502 3187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:55:37.470738 3187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:55:37.494814 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.484263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:35971 (local address 127.0.0.1:53088)
0807 02:55:37.486394 (+ 2131us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:37.486410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:37.486536 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:37.487255 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:37.487262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:37.487290 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:37.487756 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.487772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.489950 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.489955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.492259 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.492272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.493673 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.493680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.493947 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.493985 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 02:55:37.494188 (+ 203us) client_negotiation.cc:240] Negotiation successful
0807 02:55:37.494390 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1998,"mutex_wait_us":111,"thread_start_us":847,"threads_started":1}
I20260807 02:55:37.494853 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.484912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.1.121.67:35971)
0807 02:55:37.485603 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:37.485612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:37.486672 (+ 1060us) server_negotiation.cc:407] Connection header received
0807 02:55:37.486775 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:37.486782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:37.487033 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:37.487255 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:37.488001 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.489767 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.492420 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.493460 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.493608 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.493891 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:37.494253 (+ 362us) server_negotiation.cc:299] Negotiation successful
0807 02:55:37.494459 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":1282,"threads_started":1}
I20260807 02:55:37.497339 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.484083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39735 (local address 127.0.0.1:39252)
0807 02:55:37.485494 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:37.485549 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:37.485784 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:37.487985 (+ 2201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:37.487994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:37.488038 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:37.488639 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.488652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.490648 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.490656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.493966 (+ 3310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.493980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.495964 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.495972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.496160 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.496196 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 02:55:37.496299 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 02:55:37.496463 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"mutex_wait_us":3,"thread_start_us":2051,"threads_started":1}
I20260807 02:55:37.497428 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.486272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.1.121.66:37645)
0807 02:55:37.487960 (+ 1688us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:37.487969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:37.488661 (+ 692us) server_negotiation.cc:407] Connection header received
0807 02:55:37.488897 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:37.488905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:37.489120 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:37.489310 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:37.490184 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.491607 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.494394 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.495953 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.496120 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.496366 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:37.496960 (+ 594us) server_negotiation.cc:299] Negotiation successful
0807 02:55:37.497152 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1516,"thread_start_us":1416,"threads_started":1}
I20260807 02:55:37.497804 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.484253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.1.121.65:39735)
0807 02:55:37.485005 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:37.485018 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:37.485663 (+ 645us) server_negotiation.cc:407] Connection header received
0807 02:55:37.486583 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:37.486598 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:37.486931 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:37.487187 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:37.488866 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.490514 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.494147 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.495726 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.495867 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.496057 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:37.497258 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0807 02:55:37.497486 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":328,"threads_started":1}
I20260807 02:55:37.498324 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.486029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:37645 (local address 127.0.0.1:41348)
0807 02:55:37.488514 (+ 2485us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:37.488536 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:37.488676 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:37.489491 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:37.489496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:37.489518 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:37.489957 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.489969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.491797 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.491803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.494208 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.494221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.496519 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.496525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.496738 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.496779 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 02:55:37.496918 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 02:55:37.497070 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2348,"thread_start_us":2178,"threads_started":1}
I20260807 02:55:37.499185 3780 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5c87e6001948da87ea59a33f037498 (DEFAULT_TABLE table=test-table [id=7ec1fc0f6d4642b196cb80f7a5a02ccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:37.500618 3780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5c87e6001948da87ea59a33f037498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:37.502228 3381 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5c87e6001948da87ea59a33f037498 (DEFAULT_TABLE table=test-table [id=7ec1fc0f6d4642b196cb80f7a5a02ccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:37.502209 3576 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5c87e6001948da87ea59a33f037498 (DEFAULT_TABLE table=test-table [id=7ec1fc0f6d4642b196cb80f7a5a02ccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:37.503907 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5c87e6001948da87ea59a33f037498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:37.503973 3576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5c87e6001948da87ea59a33f037498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:37.515940 3906 tablet_bootstrap.cc:492] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: Bootstrap starting.
I20260807 02:55:37.521013 3907 tablet_bootstrap.cc:492] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: Bootstrap starting.
I20260807 02:55:37.521776 3906 tablet_bootstrap.cc:654] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:37.523360 3906 log.cc:824] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:37.525367 3907 tablet_bootstrap.cc:654] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:37.528122 3907 log.cc:824] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:37.530687 3906 tablet_bootstrap.cc:492] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: No bootstrap required, opened a new log
I20260807 02:55:37.531091 3906 ts_tablet_manager.cc:1397] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260807 02:55:37.533845 3909 tablet_bootstrap.cc:492] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: Bootstrap starting.
I20260807 02:55:37.537503 3907 tablet_bootstrap.cc:492] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: No bootstrap required, opened a new log
I20260807 02:55:37.538010 3907 ts_tablet_manager.cc:1397] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20260807 02:55:37.539088 3909 tablet_bootstrap.cc:654] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:37.540782 3909 log.cc:824] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:37.547751 3906 raft_consensus.cc:348] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } }
I20260807 02:55:37.548408 3906 raft_consensus.cc:374] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:37.548571 3906 raft_consensus.cc:729] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1599b002834155b469a7796f2add9e, State: Initialized, Role: FOLLOWER
I20260807 02:55:37.549346 3907 raft_consensus.cc:348] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } }
I20260807 02:55:37.549754 3907 raft_consensus.cc:374] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:37.549890 3907 raft_consensus.cc:729] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e816245f36447ebceb5b4ddf61675a, State: Initialized, Role: FOLLOWER
I20260807 02:55:37.549605 3906 consensus_queue.cc:260] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } }
I20260807 02:55:37.550576 3907 consensus_queue.cc:260] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } }
I20260807 02:55:37.551617 3877 heartbeater.cc:503] Master 127.1.121.126:41433 was elected leader, sending a full tablet report...
I20260807 02:55:37.552877 3907 ts_tablet_manager.cc:1428] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260807 02:55:37.553282 3906 ts_tablet_manager.cc:1428] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: Time spent starting tablet: real 0.022s user 0.021s sys 0.000s
I20260807 02:55:37.555202 3909 tablet_bootstrap.cc:492] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: No bootstrap required, opened a new log
I20260807 02:55:37.555615 3909 ts_tablet_manager.cc:1397] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20260807 02:55:37.564147 3909 raft_consensus.cc:348] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } }
I20260807 02:55:37.564553 3909 raft_consensus.cc:374] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:37.564663 3909 raft_consensus.cc:729] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3218d94b5b434de18c2bd66536df1390, State: Initialized, Role: FOLLOWER
I20260807 02:55:37.565364 3909 consensus_queue.cc:260] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } }
I20260807 02:55:37.566612 3680 heartbeater.cc:503] Master 127.1.121.126:41433 was elected leader, sending a full tablet report...
I20260807 02:55:37.567355 3909 ts_tablet_manager.cc:1428] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260807 02:55:37.613023 3916 raft_consensus.cc:482] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:37.613404 3916 raft_consensus.cc:504] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } }
I20260807 02:55:37.615327 3916 leader_election.cc:288] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39e816245f36447ebceb5b4ddf61675a (127.1.121.65:39735), 3218d94b5b434de18c2bd66536df1390 (127.1.121.66:37645)
I20260807 02:55:37.622234 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.615980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:43601 (local address 127.1.121.65:39735)
0807 02:55:37.616188 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:37.616192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:37.616331 (+ 139us) server_negotiation.cc:407] Connection header received
0807 02:55:37.616627 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:37.616633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:37.616780 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:37.616961 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:37.617455 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.618410 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.620218 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.621201 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.621394 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.621477 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:37.621815 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 02:55:37.621956 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:55:37.622255 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.615762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39735 (local address 127.1.121.67:43601)
0807 02:55:37.616210 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:37.616235 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:37.616418 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:37.616996 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:37.617002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:37.617036 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:37.617275 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.617285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.618581 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.618588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.620020 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.620034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.621379 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.621386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.621612 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.621639 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 02:55:37.621784 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 02:55:37.621935 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":1}
I20260807 02:55:37.622488 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.615887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:41447 (local address 127.1.121.66:37645)
0807 02:55:37.616100 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:37.616107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:37.617108 (+ 1001us) server_negotiation.cc:407] Connection header received
0807 02:55:37.617430 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:37.617435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:37.617492 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:37.617601 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:37.618172 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.618924 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.620667 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.621508 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.621750 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.621826 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:37.622086 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 02:55:37.622212 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 02:55:37.623049 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.615870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:37645 (local address 127.1.121.67:41447)
0807 02:55:37.616956 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:37.616997 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:37.617265 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:37.617617 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:37.617621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:37.617645 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:37.618007 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.618016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.619073 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.619078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.620496 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:37.620508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.621750 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.621757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:37.621930 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.621953 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:55:37.622058 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 02:55:37.622198 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":1,"thread_start_us":613,"threads_started":1}
I20260807 02:55:37.623623 3408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5c87e6001948da87ea59a33f037498" candidate_uuid: "fe1599b002834155b469a7796f2add9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e816245f36447ebceb5b4ddf61675a" is_pre_election: true
I20260807 02:55:37.624063 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5c87e6001948da87ea59a33f037498" candidate_uuid: "fe1599b002834155b469a7796f2add9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3218d94b5b434de18c2bd66536df1390" is_pre_election: true
I20260807 02:55:37.624207 3408 raft_consensus.cc:2454] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe1599b002834155b469a7796f2add9e in term 0.
I20260807 02:55:37.624547 3613 raft_consensus.cc:2454] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe1599b002834155b469a7796f2add9e in term 0.
I20260807 02:55:37.625041 3711 leader_election.cc:302] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e816245f36447ebceb5b4ddf61675a, fe1599b002834155b469a7796f2add9e; no voters:
I20260807 02:55:37.625478 3916 raft_consensus.cc:2781] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:55:37.625666 3916 raft_consensus.cc:482] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:55:37.625789 3916 raft_consensus.cc:3037] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:37.628383 3916 raft_consensus.cc:504] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } }
I20260807 02:55:37.629134 3916 leader_election.cc:288] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [CANDIDATE]: Term 1 election: Requested vote from peers 39e816245f36447ebceb5b4ddf61675a (127.1.121.65:39735), 3218d94b5b434de18c2bd66536df1390 (127.1.121.66:37645)
I20260807 02:55:37.629532 3408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5c87e6001948da87ea59a33f037498" candidate_uuid: "fe1599b002834155b469a7796f2add9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e816245f36447ebceb5b4ddf61675a"
I20260807 02:55:37.629634 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5c87e6001948da87ea59a33f037498" candidate_uuid: "fe1599b002834155b469a7796f2add9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3218d94b5b434de18c2bd66536df1390"
I20260807 02:55:37.629763 3408 raft_consensus.cc:3037] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:37.629901 3613 raft_consensus.cc:3037] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:37.632300 3613 raft_consensus.cc:2454] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe1599b002834155b469a7796f2add9e in term 1.
I20260807 02:55:37.632627 3408 raft_consensus.cc:2454] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe1599b002834155b469a7796f2add9e in term 1.
I20260807 02:55:37.632717 3708 leader_election.cc:302] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3218d94b5b434de18c2bd66536df1390, fe1599b002834155b469a7796f2add9e; no voters:
I20260807 02:55:37.633126 3916 raft_consensus.cc:2781] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:37.633888 3916 raft_consensus.cc:686] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 1 LEADER]: Becoming Leader. State: Replica: fe1599b002834155b469a7796f2add9e, State: Running, Role: LEADER
I20260807 02:55:37.634403 3916 consensus_queue.cc:237] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } }
I20260807 02:55:37.639539 3188 catalog_manager.cc:5495] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e reported cstate change: term changed from 0 to 1, leader changed from <none> to fe1599b002834155b469a7796f2add9e (127.1.121.67). New cstate: current_term: 1 leader_uuid: "fe1599b002834155b469a7796f2add9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } health_report { overall_health: UNKNOWN } } }
W20260807 02:55:37.640240 3878 tablet.cc:2369] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:55:37.702373 3480 tablet.cc:2369] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:55:37.704375 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:37.697992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.1.121.67:35971)
0807 02:55:37.698189 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:37.698194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:37.698474 (+ 280us) server_negotiation.cc:407] Connection header received
0807 02:55:37.698893 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:37.698897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:37.698974 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:37.699089 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:37.699464 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.700158 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.701762 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:37.702185 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:37.702277 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:37.703060 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:37.703739 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:37.703854 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:37.704005 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 02:55:37.704075 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:55:37.714699 3613 raft_consensus.cc:1261] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Refusing update from remote peer fe1599b002834155b469a7796f2add9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:37.715749 3408 raft_consensus.cc:1261] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [term 1 FOLLOWER]: Refusing update from remote peer fe1599b002834155b469a7796f2add9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:37.716076 3920 consensus_queue.cc:1035] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:37.716686 3916 consensus_queue.cc:1035] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:37.754840 3187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:37.755285 3187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:37.759711 3187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5: Sending DeleteTablet for 3 replicas of tablet 7e5c87e6001948da87ea59a33f037498
I20260807 02:55:37.761545 3381 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5c87e6001948da87ea59a33f037498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:37 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:39252
I20260807 02:55:37.761797 3576 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5c87e6001948da87ea59a33f037498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:37 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:41348
I20260807 02:55:37.762715 3780 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5c87e6001948da87ea59a33f037498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:37 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:53088
I20260807 02:55:37.763707 3187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:55:37.764824 3187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:55:37.767165 3943 tablet_replica.cc:330] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: stopping tablet replica
I20260807 02:55:37.768148 3943 raft_consensus.cc:2229] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:55:37.769028 3942 tablet_replica.cc:330] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: stopping tablet replica
I20260807 02:55:37.770437 3942 raft_consensus.cc:2229] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:37.769966 3943 raft_consensus.cc:2258] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:37.771602 3942 raft_consensus.cc:2258] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:37.774801 3942 ts_tablet_manager.cc:1905] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:37.776437 3380 tablet_service.cc:1467] Processing CreateTablet for tablet c33ea32bcbb24e70b54dc98d7f2e0bb8 (DEFAULT_TABLE table=test-table [id=385ad9194109465797a145744767ba30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:37.777155 3941 tablet_replica.cc:330] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: stopping tablet replica
I20260807 02:55:37.777247 3380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33ea32bcbb24e70b54dc98d7f2e0bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:37.777623 3941 raft_consensus.cc:2229] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:37.778142 3941 raft_consensus.cc:2258] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:37.778918 3943 ts_tablet_manager.cc:1905] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:37.782501 3576 tablet_service.cc:1467] Processing CreateTablet for tablet c33ea32bcbb24e70b54dc98d7f2e0bb8 (DEFAULT_TABLE table=test-table [id=385ad9194109465797a145744767ba30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:37.782979 3576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33ea32bcbb24e70b54dc98d7f2e0bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:37.783923 3942 ts_tablet_manager.cc:1918] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:55:37.784205 3942 log.cc:1196] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/7e5c87e6001948da87ea59a33f037498
I20260807 02:55:37.785200 3780 tablet_service.cc:1467] Processing CreateTablet for tablet c33ea32bcbb24e70b54dc98d7f2e0bb8 (DEFAULT_TABLE table=test-table [id=385ad9194109465797a145744767ba30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:37.784930 3942 ts_tablet_manager.cc:1939] T 7e5c87e6001948da87ea59a33f037498 P 3218d94b5b434de18c2bd66536df1390: Deleting consensus metadata
I20260807 02:55:37.785745 3780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33ea32bcbb24e70b54dc98d7f2e0bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:37.786810 3941 ts_tablet_manager.cc:1905] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:37.795724 3943 ts_tablet_manager.cc:1918] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:55:37.796010 3943 log.cc:1196] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/7e5c87e6001948da87ea59a33f037498
I20260807 02:55:37.796909 3941 ts_tablet_manager.cc:1918] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:55:37.797266 3941 log.cc:1196] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/7e5c87e6001948da87ea59a33f037498
I20260807 02:55:37.797329 3943 ts_tablet_manager.cc:1939] T 7e5c87e6001948da87ea59a33f037498 P fe1599b002834155b469a7796f2add9e: Deleting consensus metadata
I20260807 02:55:37.798290 3941 ts_tablet_manager.cc:1939] T 7e5c87e6001948da87ea59a33f037498 P 39e816245f36447ebceb5b4ddf61675a: Deleting consensus metadata
I20260807 02:55:37.800042 3169 catalog_manager.cc:4841] TS fe1599b002834155b469a7796f2add9e (127.1.121.67:35971): tablet 7e5c87e6001948da87ea59a33f037498 (table test-table [id=7ec1fc0f6d4642b196cb80f7a5a02ccc]) successfully deleted
I20260807 02:55:37.800940 3174 catalog_manager.cc:4841] TS 39e816245f36447ebceb5b4ddf61675a (127.1.121.65:39735): tablet 7e5c87e6001948da87ea59a33f037498 (table test-table [id=7ec1fc0f6d4642b196cb80f7a5a02ccc]) successfully deleted
I20260807 02:55:37.801967 3169 catalog_manager.cc:4841] TS 3218d94b5b434de18c2bd66536df1390 (127.1.121.66:37645): tablet 7e5c87e6001948da87ea59a33f037498 (table test-table [id=7ec1fc0f6d4642b196cb80f7a5a02ccc]) successfully deleted
I20260807 02:55:37.805860 3909 tablet_bootstrap.cc:492] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: Bootstrap starting.
I20260807 02:55:37.808974 3909 tablet_bootstrap.cc:654] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:37.809803 3907 tablet_bootstrap.cc:492] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: Bootstrap starting.
I20260807 02:55:37.815459 3907 tablet_bootstrap.cc:654] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:37.824884 3906 tablet_bootstrap.cc:492] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: Bootstrap starting.
I20260807 02:55:37.828174 3906 tablet_bootstrap.cc:654] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:37.829456 3907 tablet_bootstrap.cc:492] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: No bootstrap required, opened a new log
I20260807 02:55:37.831596 3907 ts_tablet_manager.cc:1397] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.004s
I20260807 02:55:37.833313 3907 raft_consensus.cc:348] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:37.833801 3907 raft_consensus.cc:374] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:37.834028 3907 raft_consensus.cc:729] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e816245f36447ebceb5b4ddf61675a, State: Initialized, Role: FOLLOWER
I20260807 02:55:37.834530 3907 consensus_queue.cc:260] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:37.836746 3909 tablet_bootstrap.cc:492] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: No bootstrap required, opened a new log
I20260807 02:55:37.837037 3909 ts_tablet_manager.cc:1397] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.014s
I20260807 02:55:37.838621 3909 raft_consensus.cc:348] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:37.838943 3909 raft_consensus.cc:374] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:37.839152 3909 raft_consensus.cc:729] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3218d94b5b434de18c2bd66536df1390, State: Initialized, Role: FOLLOWER
I20260807 02:55:37.839552 3909 consensus_queue.cc:260] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:37.839776 3907 ts_tablet_manager.cc:1428] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260807 02:55:37.840123 3906 tablet_bootstrap.cc:492] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: No bootstrap required, opened a new log
I20260807 02:55:37.840463 3906 ts_tablet_manager.cc:1397] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20260807 02:55:37.841789 3906 raft_consensus.cc:348] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:37.842324 3906 raft_consensus.cc:374] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:37.842561 3906 raft_consensus.cc:729] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1599b002834155b469a7796f2add9e, State: Initialized, Role: FOLLOWER
I20260807 02:55:37.842994 3906 consensus_queue.cc:260] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:37.844399 3909 ts_tablet_manager.cc:1428] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260807 02:55:37.844429 3906 ts_tablet_manager.cc:1428] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260807 02:55:38.040455 3681 tablet.cc:2369] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:55:38.123448 3917 raft_consensus.cc:482] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:38.123698 3917 raft_consensus.cc:504] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.125303 3917 leader_election.cc:288] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3218d94b5b434de18c2bd66536df1390 (127.1.121.66:37645), fe1599b002834155b469a7796f2add9e (127.1.121.67:35971)
I20260807 02:55:38.135093 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.126617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:55109 (local address 127.1.121.67:35971)
0807 02:55:38.126810 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.126816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.127516 (+ 700us) server_negotiation.cc:407] Connection header received
0807 02:55:38.127847 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.127855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.127983 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.128123 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:38.129180 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.130594 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.132760 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.134045 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.134212 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.134311 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.134605 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.134760 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:55:38.135175 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.126480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:35971 (local address 127.1.121.65:55109)
0807 02:55:38.127377 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:38.127396 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:38.127650 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:38.128164 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:38.128169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:38.128197 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:38.128989 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.129005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.130733 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.130738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.132582 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.132594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.134232 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.134238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.134401 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.134451 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 02:55:38.134565 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 02:55:38.134715 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":1,"thread_start_us":565,"threads_started":1}
I20260807 02:55:38.136236 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.125627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:37645 (local address 127.1.121.65:44337)
0807 02:55:38.126246 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:38.126267 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:38.126412 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:38.127415 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:38.127422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:38.127452 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:38.128561 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.128578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.130417 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.130425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.133302 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.133318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.135020 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.135029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.135616 (+ 587us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.135652 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 02:55:38.135733 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 02:55:38.135864 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":342,"threads_started":1}
I20260807 02:55:38.136250 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.125721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:44337 (local address 127.1.121.66:37645)
0807 02:55:38.127056 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.127065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.127087 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:55:38.127176 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.127181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.127259 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.127371 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:38.128893 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.130236 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.133491 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.134749 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.135639 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.135733 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.135782 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.135907 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1179,"thread_start_us":1100,"threads_started":1}
I20260807 02:55:38.136344 3806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33ea32bcbb24e70b54dc98d7f2e0bb8" candidate_uuid: "39e816245f36447ebceb5b4ddf61675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1599b002834155b469a7796f2add9e" is_pre_election: true
I20260807 02:55:38.136907 3806 raft_consensus.cc:2454] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e816245f36447ebceb5b4ddf61675a in term 0.
I20260807 02:55:38.137291 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33ea32bcbb24e70b54dc98d7f2e0bb8" candidate_uuid: "39e816245f36447ebceb5b4ddf61675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3218d94b5b434de18c2bd66536df1390" is_pre_election: true
I20260807 02:55:38.137552 3613 raft_consensus.cc:2454] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39e816245f36447ebceb5b4ddf61675a in term 0.
I20260807 02:55:38.137633 3314 leader_election.cc:302] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e816245f36447ebceb5b4ddf61675a, fe1599b002834155b469a7796f2add9e; no voters:
I20260807 02:55:38.138159 3917 raft_consensus.cc:2781] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:55:38.138326 3917 raft_consensus.cc:482] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:55:38.138471 3917 raft_consensus.cc:3037] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:38.141016 3917 raft_consensus.cc:504] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.141690 3917 leader_election.cc:288] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [CANDIDATE]: Term 1 election: Requested vote from peers 3218d94b5b434de18c2bd66536df1390 (127.1.121.66:37645), fe1599b002834155b469a7796f2add9e (127.1.121.67:35971)
I20260807 02:55:38.142287 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33ea32bcbb24e70b54dc98d7f2e0bb8" candidate_uuid: "39e816245f36447ebceb5b4ddf61675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3218d94b5b434de18c2bd66536df1390"
I20260807 02:55:38.142393 3806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33ea32bcbb24e70b54dc98d7f2e0bb8" candidate_uuid: "39e816245f36447ebceb5b4ddf61675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1599b002834155b469a7796f2add9e"
I20260807 02:55:38.142527 3613 raft_consensus.cc:3037] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:38.142690 3806 raft_consensus.cc:3037] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:38.144712 3613 raft_consensus.cc:2454] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e816245f36447ebceb5b4ddf61675a in term 1.
I20260807 02:55:38.145242 3314 leader_election.cc:302] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3218d94b5b434de18c2bd66536df1390, 39e816245f36447ebceb5b4ddf61675a; no voters:
I20260807 02:55:38.145537 3917 raft_consensus.cc:2781] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:38.146123 3917 raft_consensus.cc:686] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 1 LEADER]: Becoming Leader. State: Replica: 39e816245f36447ebceb5b4ddf61675a, State: Running, Role: LEADER
I20260807 02:55:38.146391 3806 raft_consensus.cc:2454] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e816245f36447ebceb5b4ddf61675a in term 1.
I20260807 02:55:38.146632 3917 consensus_queue.cc:237] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.150825 3186 catalog_manager.cc:5495] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a reported cstate change: term changed from 0 to 1, leader changed from <none> to 39e816245f36447ebceb5b4ddf61675a (127.1.121.65). New cstate: current_term: 1 leader_uuid: "39e816245f36447ebceb5b4ddf61675a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } health_report { overall_health: UNKNOWN } } }
Aug 07 02:55:38 dist-test-slave-3h5h krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:38.219683 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.199800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41433 (local address 127.0.0.1:54792)
0807 02:55:38.200959 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:38.200978 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:38.201139 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:38.201599 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:38.201604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:38.202542 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:38.202836 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.202848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.204573 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.204578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.206324 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.206339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.207305 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.207316 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.207527 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.208459 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:38.208484 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:38.211911 (+ 3427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:38.216312 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:38.216330 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:38.216338 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:38.217083 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:38.217521 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:38.217527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:38.217532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:38.217800 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:38.218485 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:38.218494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:38.218680 (+ 186us) client_negotiation.cc:769] Sending connection context
0807 02:55:38.218860 (+ 180us) client_negotiation.cc:240] Negotiation successful
0807 02:55:38.219159 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":892,"threads_started":1}
I20260807 02:55:38.219846 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.199738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.1.121.126:41433)
0807 02:55:38.200462 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.200469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.201084 (+ 615us) server_negotiation.cc:407] Connection header received
0807 02:55:38.201278 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.201284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.201368 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.201496 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:38.203018 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.204365 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.206481 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.207114 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.207234 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.212120 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:38.212127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:38.212132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:38.212169 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:38.216156 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.217231 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.217237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.217241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.217381 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.217948 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.217954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.217958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.218328 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:38.218491 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.219018 (+ 527us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.219342 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":587,"thread_start_us":402,"threads_started":1}
I20260807 02:55:38.226507 3186 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:54792:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 02:55:38.227782 3186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:55:38.235265 3380 tablet_service.cc:1467] Processing CreateTablet for tablet d38d33c7ffe24ddd8cb342662b188da0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78672294ce4b4d7ca46c7f7a2faf30e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:55:38.235656 3780 tablet_service.cc:1467] Processing CreateTablet for tablet d38d33c7ffe24ddd8cb342662b188da0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78672294ce4b4d7ca46c7f7a2faf30e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:55:38.235932 3380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38d33c7ffe24ddd8cb342662b188da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:38.236217 3780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38d33c7ffe24ddd8cb342662b188da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:38.239601 3576 tablet_service.cc:1467] Processing CreateTablet for tablet d38d33c7ffe24ddd8cb342662b188da0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78672294ce4b4d7ca46c7f7a2faf30e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:55:38.240259 3576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38d33c7ffe24ddd8cb342662b188da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:38.242444 3906 tablet_bootstrap.cc:492] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: Bootstrap starting.
I20260807 02:55:38.244146 3906 tablet_bootstrap.cc:654] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:38.246547 3906 tablet_bootstrap.cc:492] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: No bootstrap required, opened a new log
I20260807 02:55:38.246757 3906 ts_tablet_manager.cc:1397] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260807 02:55:38.247435 3906 raft_consensus.cc:348] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.247671 3906 raft_consensus.cc:374] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:38.247776 3906 raft_consensus.cc:729] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1599b002834155b469a7796f2add9e, State: Initialized, Role: FOLLOWER
I20260807 02:55:38.248036 3906 consensus_queue.cc:260] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.248703 3934 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.249008 3934 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:38.251492 3906 ts_tablet_manager.cc:1428] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260807 02:55:38.253610 3909 tablet_bootstrap.cc:492] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: Bootstrap starting.
I20260807 02:55:38.255857 3909 tablet_bootstrap.cc:654] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:38.257447 3907 tablet_bootstrap.cc:492] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: Bootstrap starting.
I20260807 02:55:38.259524 3907 tablet_bootstrap.cc:654] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:38.262383 3907 tablet_bootstrap.cc:492] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: No bootstrap required, opened a new log
I20260807 02:55:38.262626 3907 ts_tablet_manager.cc:1397] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260807 02:55:38.263415 3907 raft_consensus.cc:348] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.263733 3907 raft_consensus.cc:374] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:38.264859 3909 tablet_bootstrap.cc:492] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: No bootstrap required, opened a new log
I20260807 02:55:38.265120 3909 ts_tablet_manager.cc:1397] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20260807 02:55:38.266124 3909 raft_consensus.cc:348] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.266396 3909 raft_consensus.cc:374] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:38.266511 3909 raft_consensus.cc:729] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3218d94b5b434de18c2bd66536df1390, State: Initialized, Role: FOLLOWER
I20260807 02:55:38.266769 3907 raft_consensus.cc:729] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e816245f36447ebceb5b4ddf61675a, State: Initialized, Role: FOLLOWER
I20260807 02:55:38.267069 3907 consensus_queue.cc:260] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.267624 3917 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.267927 3917 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:38.268481 3907 ts_tablet_manager.cc:1428] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260807 02:55:38.268919 3909 consensus_queue.cc:260] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.269501 3918 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.269979 3918 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:38.271570 3909 ts_tablet_manager.cc:1428] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260807 02:55:38.399725 3934 raft_consensus.cc:482] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:38.400056 3934 raft_consensus.cc:504] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.401348 3934 leader_election.cc:288] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39e816245f36447ebceb5b4ddf61675a (127.1.121.65:39735), 3218d94b5b434de18c2bd66536df1390 (127.1.121.66:37645)
I20260807 02:55:38.401752 3408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38d33c7ffe24ddd8cb342662b188da0" candidate_uuid: "fe1599b002834155b469a7796f2add9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e816245f36447ebceb5b4ddf61675a" is_pre_election: true
I20260807 02:55:38.401731 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38d33c7ffe24ddd8cb342662b188da0" candidate_uuid: "fe1599b002834155b469a7796f2add9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3218d94b5b434de18c2bd66536df1390" is_pre_election: true
I20260807 02:55:38.402005 3408 raft_consensus.cc:2454] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe1599b002834155b469a7796f2add9e in term 0.
I20260807 02:55:38.402029 3613 raft_consensus.cc:2454] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe1599b002834155b469a7796f2add9e in term 0.
I20260807 02:55:38.402475 3708 leader_election.cc:302] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3218d94b5b434de18c2bd66536df1390, fe1599b002834155b469a7796f2add9e; no voters:
I20260807 02:55:38.402841 3934 raft_consensus.cc:2781] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:55:38.402976 3934 raft_consensus.cc:482] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:55:38.403090 3934 raft_consensus.cc:3037] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:38.405637 3934 raft_consensus.cc:504] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.406279 3934 leader_election.cc:288] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [CANDIDATE]: Term 1 election: Requested vote from peers 39e816245f36447ebceb5b4ddf61675a (127.1.121.65:39735), 3218d94b5b434de18c2bd66536df1390 (127.1.121.66:37645)
I20260807 02:55:38.406800 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38d33c7ffe24ddd8cb342662b188da0" candidate_uuid: "fe1599b002834155b469a7796f2add9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3218d94b5b434de18c2bd66536df1390"
I20260807 02:55:38.406912 3408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38d33c7ffe24ddd8cb342662b188da0" candidate_uuid: "fe1599b002834155b469a7796f2add9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e816245f36447ebceb5b4ddf61675a"
I20260807 02:55:38.407065 3613 raft_consensus.cc:3037] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:38.407114 3408 raft_consensus.cc:3037] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:38.410223 3613 raft_consensus.cc:2454] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe1599b002834155b469a7796f2add9e in term 1.
I20260807 02:55:38.410233 3408 raft_consensus.cc:2454] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe1599b002834155b469a7796f2add9e in term 1.
I20260807 02:55:38.410770 3711 leader_election.cc:302] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e816245f36447ebceb5b4ddf61675a, fe1599b002834155b469a7796f2add9e; no voters:
I20260807 02:55:38.411263 3934 raft_consensus.cc:2781] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:38.411839 3934 raft_consensus.cc:686] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [term 1 LEADER]: Becoming Leader. State: Replica: fe1599b002834155b469a7796f2add9e, State: Running, Role: LEADER
I20260807 02:55:38.412295 3934 consensus_queue.cc:237] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } }
I20260807 02:55:38.414232 3964 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: TxnStatusTablet state changed. Reason: New leader fe1599b002834155b469a7796f2add9e. Latest consensus state: current_term: 1 leader_uuid: "fe1599b002834155b469a7796f2add9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.416183 3964 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:55:38.415997 3186 catalog_manager.cc:5495] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e reported cstate change: term changed from 0 to 1, leader changed from <none> to fe1599b002834155b469a7796f2add9e (127.1.121.67). New cstate: current_term: 1 leader_uuid: "fe1599b002834155b469a7796f2add9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } health_report { overall_health: HEALTHY } } }
I20260807 02:55:38.422228 3967 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 02:55:38.422506 3967 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 02:55:38.451756 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.445770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.1.121.67:35971)
0807 02:55:38.445963 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.445968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.446039 (+ 71us) server_negotiation.cc:407] Connection header received
0807 02:55:38.446342 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.446345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.446410 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.446491 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:38.447007 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.447748 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.449941 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.450385 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.450477 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.451083 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:38.451183 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:38.451253 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.451450 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.451516 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 02:55:38.452031 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.445618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:35971 (local address 127.0.0.1:53102)
0807 02:55:38.445933 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:38.445958 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:38.446130 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:38.446572 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:38.446578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:38.446595 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:55:38.446866 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.446878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.447898 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.447905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.449795 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.449813 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.450561 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.450568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.450808 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.450899 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:55:38.451296 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:55:38.451319 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:55:38.451431 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 02:55:38.451619 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 02:55:38.457273 3408 raft_consensus.cc:1261] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a [term 1 FOLLOWER]: Refusing update from remote peer fe1599b002834155b469a7796f2add9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:38.457559 3613 raft_consensus.cc:1261] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Refusing update from remote peer fe1599b002834155b469a7796f2add9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:38.457968 3964 consensus_queue.cc:1035] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:38.458273 3934 consensus_queue.cc:1035] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:38.464391 3917 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: TxnStatusTablet state changed. Reason: New leader fe1599b002834155b469a7796f2add9e. Latest consensus state: current_term: 1 leader_uuid: "fe1599b002834155b469a7796f2add9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.464795 3917 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:38.466454 3918 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: TxnStatusTablet state changed. Reason: New leader fe1599b002834155b469a7796f2add9e. Latest consensus state: current_term: 1 leader_uuid: "fe1599b002834155b469a7796f2add9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.467336 3918 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:38.469676 3972 mvcc.cc:205] Tried to move back new op lower bound from 7315748202314141696 to 7315748202139750400. Current Snapshot: MvccSnapshot[applied={T|T < 7315748202314141696}]
I20260807 02:55:38.471148 3964 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe1599b002834155b469a7796f2add9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.471205 3934 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe1599b002834155b469a7796f2add9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.471436 3934 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:55:38.471443 3964 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P fe1599b002834155b469a7796f2add9e: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:55:38.474030 3971 mvcc.cc:205] Tried to move back new op lower bound from 7315748202314141696 to 7315748202139750400. Current Snapshot: MvccSnapshot[applied={T|T < 7315748202314141696 or (T in {7315748202314141696})}]
I20260807 02:55:38.476727 3917 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe1599b002834155b469a7796f2add9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.477085 3917 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P 39e816245f36447ebceb5b4ddf61675a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:38.489679 3918 tablet_replica.cc:439] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe1599b002834155b469a7796f2add9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e816245f36447ebceb5b4ddf61675a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39735 } } peers { permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 } } peers { permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 } } }
I20260807 02:55:38.490049 3918 tablet_replica.cc:442] T d38d33c7ffe24ddd8cb342662b188da0 P 3218d94b5b434de18c2bd66536df1390: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:38.500370 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.482216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.1.121.65:39735)
0807 02:55:38.493778 (+ 11562us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.493785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.493812 (+ 27us) server_negotiation.cc:407] Connection header received
0807 02:55:38.493929 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.493935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.494059 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.494261 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:38.494753 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.495919 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.497575 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.498273 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.498411 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.498839 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:38.499669 (+ 830us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:38.499821 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.499882 (+ 61us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.499979 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11392,"thread_start_us":11490,"threads_started":1}
I20260807 02:55:38.520141 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.513441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:54021 (local address 127.1.121.67:35971)
0807 02:55:38.513666 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.513674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.514305 (+ 631us) server_negotiation.cc:407] Connection header received
0807 02:55:38.514480 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.514486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.514585 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.514691 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:38.515174 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.516419 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.517920 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.518674 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.518817 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.519206 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:38.519364 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:38.519456 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.519648 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.519735 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:55:38.520123 3989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.513429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:35971 (local address 127.1.121.65:54021)
0807 02:55:38.514202 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:38.514219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:38.514344 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:38.514711 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:38.514714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:38.514723 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:55:38.515009 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.515017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.516559 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.516562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.517779 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.517792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.518851 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.518856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.518966 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.519020 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:55:38.519509 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:55:38.519520 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 02:55:38.519602 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 02:55:38.519742 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":426,"threads_started":1}
I20260807 02:55:38.535579 3613 raft_consensus.cc:1261] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Refusing update from remote peer 39e816245f36447ebceb5b4ddf61675a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:38.535743 3806 raft_consensus.cc:1261] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [term 1 FOLLOWER]: Refusing update from remote peer 39e816245f36447ebceb5b4ddf61675a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:38.536608 3951 consensus_queue.cc:1035] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe1599b002834155b469a7796f2add9e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 35971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:38.537066 3917 consensus_queue.cc:1035] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3218d94b5b434de18c2bd66536df1390" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:38.591674 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.584031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:46415 (local address 127.1.121.65:39735)
0807 02:55:38.584272 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.584279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.584729 (+ 450us) server_negotiation.cc:407] Connection header received
0807 02:55:38.584985 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.584991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.585183 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.585306 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:38.586048 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.587339 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.589140 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.589885 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.590039 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.590882 (+ 843us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:38.591055 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:38.591160 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.591344 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.591415 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 02:55:38.591970 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.583880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39735 (local address 127.1.121.67:46415)
0807 02:55:38.584594 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:38.584617 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:38.584791 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:38.585324 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:38.585329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:38.585343 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:55:38.585814 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.585834 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.587505 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.587515 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.588976 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:38.588995 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.590221 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.590229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:38.590453 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.590564 (+ 111us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:55:38.591187 (+ 623us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:55:38.591212 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 02:55:38.591307 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 02:55:38.591477 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":323,"threads_started":1}
I20260807 02:55:38.638514 3186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:38.638854 3186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:38.643110 3186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 456f3fe7048e4d558476f6f75e814fd5: Sending DeleteTablet for 3 replicas of tablet c33ea32bcbb24e70b54dc98d7f2e0bb8
I20260807 02:55:38.644366 3576 tablet_service.cc:1514] Processing DeleteTablet for tablet c33ea32bcbb24e70b54dc98d7f2e0bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:38 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:41348
I20260807 02:55:38.644927 3780 tablet_service.cc:1514] Processing DeleteTablet for tablet c33ea32bcbb24e70b54dc98d7f2e0bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:38 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:53088
I20260807 02:55:38.645046 3380 tablet_service.cc:1514] Processing DeleteTablet for tablet c33ea32bcbb24e70b54dc98d7f2e0bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:38 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:39252
I20260807 02:55:38.657225 4000 tablet_replica.cc:330] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: stopping tablet replica
I20260807 02:55:38.657501 4000 raft_consensus.cc:2229] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:38.657841 4000 raft_consensus.cc:2258] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:38.659523 4000 ts_tablet_manager.cc:1905] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:38.668628 4003 tablet_replica.cc:330] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: stopping tablet replica
I20260807 02:55:38.669524 4003 raft_consensus.cc:2229] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:55:38.670660 4003 raft_consensus.cc:2258] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:38.671900 4000 ts_tablet_manager.cc:1918] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:55:38.672077 4000 log.cc:1196] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/c33ea32bcbb24e70b54dc98d7f2e0bb8
I20260807 02:55:38.672411 4003 ts_tablet_manager.cc:1905] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:38.672668 4000 ts_tablet_manager.cc:1939] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 3218d94b5b434de18c2bd66536df1390: Deleting consensus metadata
I20260807 02:55:38.673936 3169 catalog_manager.cc:4841] TS 3218d94b5b434de18c2bd66536df1390 (127.1.121.66:37645): tablet c33ea32bcbb24e70b54dc98d7f2e0bb8 (table test-table [id=385ad9194109465797a145744767ba30]) successfully deleted
I20260807 02:55:38.676038 4001 tablet_replica.cc:330] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: stopping tablet replica
I20260807 02:55:38.676415 4001 raft_consensus.cc:2229] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:38.677079 4001 raft_consensus.cc:2258] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:38.678325 4001 ts_tablet_manager.cc:1905] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:38.684180 4003 ts_tablet_manager.cc:1918] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:55:38.684437 4003 log.cc:1196] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/c33ea32bcbb24e70b54dc98d7f2e0bb8
I20260807 02:55:38.685935 4003 ts_tablet_manager.cc:1939] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P 39e816245f36447ebceb5b4ddf61675a: Deleting consensus metadata
I20260807 02:55:38.687487 3174 catalog_manager.cc:4841] TS 39e816245f36447ebceb5b4ddf61675a (127.1.121.65:39735): tablet c33ea32bcbb24e70b54dc98d7f2e0bb8 (table test-table [id=385ad9194109465797a145744767ba30]) successfully deleted
I20260807 02:55:38.696339 4001 ts_tablet_manager.cc:1918] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:55:38.696758 4001 log.cc:1196] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/c33ea32bcbb24e70b54dc98d7f2e0bb8
I20260807 02:55:38.697705 4001 ts_tablet_manager.cc:1939] T c33ea32bcbb24e70b54dc98d7f2e0bb8 P fe1599b002834155b469a7796f2add9e: Deleting consensus metadata
I20260807 02:55:38.699203 3169 catalog_manager.cc:4841] TS fe1599b002834155b469a7796f2add9e (127.1.121.67:35971): tablet c33ea32bcbb24e70b54dc98d7f2e0bb8 (table test-table [id=385ad9194109465797a145744767ba30]) successfully deleted
Aug 07 02:55:38 dist-test-slave-3h5h krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:55:38.715654 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.674764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.1.121.65:39735)
0807 02:55:38.674990 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.674996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.685689 (+ 10693us) server_negotiation.cc:407] Connection header received
0807 02:55:38.686227 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.686234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.686325 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.686447 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:38.691614 (+ 5167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.693432 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.696704 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.697760 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.697891 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.705704 (+ 7813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:38.705728 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:38.705751 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:38.705834 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:38.711906 (+ 6072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.712885 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.712897 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.712900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.713058 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.713654 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.713660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.713664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.714180 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:38.714558 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.714955 (+ 397us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.715201 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260807 02:55:38.716639 3473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39272
I20260807 02:55:38.748317 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.722310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.1.121.126:41433)
0807 02:55:38.729769 (+ 7459us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.729779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.729802 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:55:38.729876 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.729882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.730084 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.730224 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:38.732381 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.734047 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.735804 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.736685 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.736873 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.740471 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:38.740481 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:38.740488 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:38.740537 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:38.744980 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.745889 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.745896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.745901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.746103 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.746622 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.746628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.746631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.746996 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:38.747264 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.747671 (+ 407us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.747922 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7287,"thread_start_us":7127,"threads_started":1}
W20260807 02:55:38.749089 3186 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54804
I20260807 02:55:38.753953 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 3291
I20260807 02:55:38.785198 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.755250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.1.121.65:39735)
0807 02:55:38.755447 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.755452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.765504 (+ 10052us) server_negotiation.cc:407] Connection header received
0807 02:55:38.765721 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.765728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.765827 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.765955 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:38.767718 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.769354 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.771645 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.772466 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.772616 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.776086 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:38.776100 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:38.776106 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:38.776155 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:38.781208 (+ 5053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.782204 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.782211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.782215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.782591 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.783084 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.783090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.783095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.783498 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:38.783814 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.784212 (+ 398us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.784531 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 02:55:38.786170 3473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39280
W20260807 02:55:38.787050 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 3291: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 02:55:38.787221 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 3291
I20260807 02:55:38.819595 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 3485
Aug 07 02:55:38 dist-test-slave-3h5h krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:55:38.838456 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.820454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.1.121.66:37645)
0807 02:55:38.821340 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.821348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.821464 (+ 116us) server_negotiation.cc:407] Connection header received
0807 02:55:38.821757 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.821764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.821862 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.822004 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:38.823319 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.824172 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.825701 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.826258 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.826417 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.831435 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:38.831451 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:38.831470 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:38.831507 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:38.835137 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.835910 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.835919 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.835922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.836074 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.836623 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.836627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.836630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.837104 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:38.837451 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.837788 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.838069 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":381,"threads_started":1}
W20260807 02:55:38.839376 3670 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33452
W20260807 02:55:38.840157 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 3485: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 02:55:38.840369 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 3485
I20260807 02:55:38.872285 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 3685
Aug 07 02:55:38 dist-test-slave-3h5h krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:55:38.892112 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:38.873326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.1.121.67:35971)
0807 02:55:38.873564 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:38.873571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:38.873596 (+ 25us) server_negotiation.cc:407] Connection header received
0807 02:55:38.873682 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:38.873704 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:38.873808 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:38.873941 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:38.875183 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.876551 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.877868 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:38.878649 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:38.878812 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:38.883603 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:38.883626 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:38.883652 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:38.883704 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:38.888461 (+ 4757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.889441 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.889452 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.889456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.889677 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:38.890174 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:38.890180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:38.890183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:38.890686 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:38.891104 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:38.891328 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 02:55:38.891635 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 02:55:38.893122 3870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46886
W20260807 02:55:38.893982 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 3685: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 02:55:38.894217 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 3685
I20260807 02:55:38.932313 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 3147
I20260807 02:55:38.933934 3247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:39.067628 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 3147
2026-08-07T02:55:39Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5602 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:55:39 dist-test-slave-3h5h krb5kdc[4017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:55:39 dist-test-slave-3h5h krb5kdc[4017](info): set up 2 sockets
Aug 07 02:55:39 dist-test-slave-3h5h krb5kdc[4017](info): commencing operation
krb5kdc: starting...
W20260807 02:55:40.393631 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:55:40 dist-test-slave-3h5h krb5kdc[4017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:55:40.624998 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33811
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:33811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:40.961692 4042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:40.962044 4042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:40.962213 4042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:40.969669 4042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:55:40.969785 4042 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 02:55:40.969874 4042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:40.970034 4042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:40.970150 4042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:55:40.970265 4042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:55:40.982158 4042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:33811
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:40.983744 4042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:40.985514 4042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:40.999972 4047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:41.000005 4048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:41.000103 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:41.001225 4042 server_base.cc:1034] running on GCE node
I20260807 02:55:41.002051 4042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:41.003677 4042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:41.005075 4042 hybrid_clock.cc:648] HybridClock initialized: now 1786071341004780 us; error 238 us; skew 500 ppm
I20260807 02:55:41.005497 4042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:41 dist-test-slave-3h5h krb5kdc[4017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:41.013082 4042 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:55:41.015005 4042 webserver.cc:460] Webserver started at http://127.1.121.126:45291/ using document root <none> and password file <none>
I20260807 02:55:41.015933 4042 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:41.016033 4042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:41.016397 4042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:41.019837 4042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "2de93b5fb8a2465192593c43f2541e71"
format_stamp: "Formatted at 2026-08-07 02:55:41 on dist-test-slave-3h5h"
server_key: "b8207aebf037b81eed89419ccae25504"
server_key_iv: "bf6205d5224574d950edba52ed8b2412"
server_key_version: "encryptionkey@0"
I20260807 02:55:41.020666 4042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "2de93b5fb8a2465192593c43f2541e71"
format_stamp: "Formatted at 2026-08-07 02:55:41 on dist-test-slave-3h5h"
server_key: "b8207aebf037b81eed89419ccae25504"
server_key_iv: "bf6205d5224574d950edba52ed8b2412"
server_key_version: "encryptionkey@0"
I20260807 02:55:41.026530 4042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 02:55:41.030849 4062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:41.032557 4042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:55:41.032809 4042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "2de93b5fb8a2465192593c43f2541e71"
format_stamp: "Formatted at 2026-08-07 02:55:41 on dist-test-slave-3h5h"
server_key: "b8207aebf037b81eed89419ccae25504"
server_key_iv: "bf6205d5224574d950edba52ed8b2412"
server_key_version: "encryptionkey@0"
I20260807 02:55:41.033162 4042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:41.054641 4042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:41.059583 4042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:41.059880 4042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:41.083866 4042 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:33811
I20260807 02:55:41.083840 4143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:33811 every 8 connection(s)
I20260807 02:55:41.085363 4042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:55:41.087028 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 4042
I20260807 02:55:41.087272 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:55:41.088163 1509 external_mini_cluster.cc:1428] Setting key 920a50c1da1d9234c7a36bb6e0c87f2e
I20260807 02:55:41.090291 4144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:41.105381 4144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71: Bootstrap starting.
Aug 07 02:55:41 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:41.108930 4144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:41.110203 4144 log.cc:824] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:41.113255 4144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71: No bootstrap required, opened a new log
I20260807 02:55:41.116297 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:41.090361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.1.121.126:33811)
0807 02:55:41.091108 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:41.091126 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:41.091171 (+ 45us) server_negotiation.cc:407] Connection header received
0807 02:55:41.093814 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:41.093845 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:41.094370 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:41.094821 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:41.096738 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:41.098969 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:41.100904 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:41.102314 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:41.102564 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:41.107737 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:41.107747 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:41.107763 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:41.107803 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:41.111779 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:41.112537 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:41.112544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:41.112551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:41.112781 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:41.113229 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:41.113234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:41.113238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:41.113874 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:41.114210 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:41.114700 (+ 490us) server_negotiation.cc:299] Negotiation successful
0807 02:55:41.115167 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":322,"threads_started":1}
I20260807 02:55:41.122125 4144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de93b5fb8a2465192593c43f2541e71" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33811 } }
I20260807 02:55:41.122496 4144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:41.122605 4144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de93b5fb8a2465192593c43f2541e71, State: Initialized, Role: FOLLOWER
I20260807 02:55:41.123148 4144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de93b5fb8a2465192593c43f2541e71" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33811 } }
I20260807 02:55:41.123392 4144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:55:41.123545 4144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:55:41.123793 4144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:41.193066 4144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de93b5fb8a2465192593c43f2541e71" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33811 } }
I20260807 02:55:41.193830 4144 leader_election.cc:302] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2de93b5fb8a2465192593c43f2541e71; no voters:
I20260807 02:55:41.194554 4144 leader_election.cc:288] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:55:41.194780 4151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:41.196002 4151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [term 1 LEADER]: Becoming Leader. State: Replica: 2de93b5fb8a2465192593c43f2541e71, State: Running, Role: LEADER
I20260807 02:55:41.196666 4151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de93b5fb8a2465192593c43f2541e71" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33811 } }
I20260807 02:55:41.197271 4144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:55:41.202801 4153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2de93b5fb8a2465192593c43f2541e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de93b5fb8a2465192593c43f2541e71" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33811 } } }
I20260807 02:55:41.203284 4153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:41.203711 4154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2de93b5fb8a2465192593c43f2541e71. Latest consensus state: current_term: 1 leader_uuid: "2de93b5fb8a2465192593c43f2541e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2de93b5fb8a2465192593c43f2541e71" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33811 } } }
I20260807 02:55:41.204002 4154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:41.205943 4165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:55:41.210556 4165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:55:41.219689 4165 catalog_manager.cc:1347] Generated new cluster ID: ada6536a68b2468ba3f233b675dfcc10
I20260807 02:55:41.219815 4165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:55:41.264782 4165 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:55:41.266806 4165 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:55:41.291525 4165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71: Generated new TSK 0
I20260807 02:55:41.292706 4165 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
W20260807 02:55:42.119134 1509 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.121.65: real 0.814s user 0.001s sys 0.015s
I20260807 02:55:42.120335 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33811
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:42.497766 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:42.498144 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:42.498351 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:42.507586 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:42.507709 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:42.507903 4177 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 02:55:42.507977 4177 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:55:42.508051 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:55:42.519961 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.121.126:33811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:42.521661 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:42.523703 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:42.538683 4184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:42.538635 4188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:42.538717 4186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:42.540099 4177 server_base.cc:1034] running on GCE node
I20260807 02:55:42.540774 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:42.541791 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:42.543048 4177 hybrid_clock.cc:648] HybridClock initialized: now 1786071342542973 us; error 64 us; skew 500 ppm
I20260807 02:55:42.543392 4177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:42 dist-test-slave-3h5h krb5kdc[4017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071342, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:42.546895 4177 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:55:42.548465 4177 webserver.cc:460] Webserver started at http://127.1.121.65:42597/ using document root <none> and password file <none>
I20260807 02:55:42.549273 4177 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:42.549424 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:42.549758 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:42.552821 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "9d1d0d70d38444e7b8729d86e1445763"
format_stamp: "Formatted at 2026-08-07 02:55:42 on dist-test-slave-3h5h"
server_key: "e4d4b649222fc38fb63131f634be2109"
server_key_iv: "a4e0f032d885a71f3953583a13cbab7d"
server_key_version: "encryptionkey@0"
I20260807 02:55:42.553766 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "9d1d0d70d38444e7b8729d86e1445763"
format_stamp: "Formatted at 2026-08-07 02:55:42 on dist-test-slave-3h5h"
server_key: "e4d4b649222fc38fb63131f634be2109"
server_key_iv: "a4e0f032d885a71f3953583a13cbab7d"
server_key_version: "encryptionkey@0"
I20260807 02:55:42.559407 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260807 02:55:42.563570 4198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:42.565214 4177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 02:55:42.565474 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "9d1d0d70d38444e7b8729d86e1445763"
format_stamp: "Formatted at 2026-08-07 02:55:42 on dist-test-slave-3h5h"
server_key: "e4d4b649222fc38fb63131f634be2109"
server_key_iv: "a4e0f032d885a71f3953583a13cbab7d"
server_key_version: "encryptionkey@0"
I20260807 02:55:42.565725 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:42.590013 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:42.595824 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:42.596200 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:42.599620 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:42.599798 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:42.600003 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:42.600163 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 07 02:55:42 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071342, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:42.629463 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:42.601523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33811 (local address 127.1.121.65:49955)
0807 02:55:42.602743 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:42.602812 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:42.606036 (+ 3224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:42.607177 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:42.607190 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:42.608321 (+ 1131us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:42.609584 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:42.609625 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:42.611130 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:42.611141 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:42.614527 (+ 3386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:42.614570 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:42.615581 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:42.615593 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:42.616023 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:42.616997 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:42.617055 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:42.621189 (+ 4134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:42.625057 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:42.625071 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:42.625103 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:42.625607 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:42.626037 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:42.626042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:42.626045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:42.626248 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:42.627010 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:42.627020 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:42.627430 (+ 410us) client_negotiation.cc:769] Sending connection context
0807 02:55:42.627706 (+ 276us) client_negotiation.cc:240] Negotiation successful
0807 02:55:42.628123 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":444,"threads_started":1}
I20260807 02:55:42.629786 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:42.600765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:49955 (local address 127.1.121.126:33811)
0807 02:55:42.601419 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:42.601426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:42.602979 (+ 1553us) server_negotiation.cc:407] Connection header received
0807 02:55:42.606544 (+ 3565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:42.606551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:42.606657 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:42.606796 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:42.609946 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:42.610889 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:42.614759 (+ 3870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:42.615252 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:42.615430 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:42.621490 (+ 6060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:42.621496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:42.621500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:42.621538 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:42.624854 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:42.625771 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:42.625774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:42.625776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:42.625904 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:42.626390 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:42.626398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:42.626404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:42.626821 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:42.628974 (+ 2153us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:42.629160 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 02:55:42.629408 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":326,"threads_started":1}
I20260807 02:55:42.658926 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:42.640292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:57075 (local address 127.1.121.126:33811)
0807 02:55:42.640517 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:42.640524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:42.641519 (+ 995us) server_negotiation.cc:407] Connection header received
0807 02:55:42.641751 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:42.641755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:42.641822 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:42.641904 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:42.643881 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:42.644961 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:42.647983 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:42.648458 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:42.648563 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:42.651425 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:42.651436 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:42.651439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:42.651473 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:42.655302 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:42.656558 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:42.656575 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:42.656580 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:42.656784 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:42.657287 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:42.657295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:42.657299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:42.657676 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:42.657850 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:42.658340 (+ 490us) server_negotiation.cc:299] Negotiation successful
0807 02:55:42.658576 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:55:42.659085 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:42.640357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33811 (local address 127.1.121.65:57075)
0807 02:55:42.641397 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:42.641425 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:42.641607 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:42.642005 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:42.642015 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:42.643084 (+ 1069us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:42.643672 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:42.643692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:42.645137 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:42.645144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:42.647791 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:42.647819 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:42.648730 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:42.648741 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:42.649023 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:42.650033 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:42.650052 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:42.651225 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:42.655466 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:42.655476 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:42.655483 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:42.656392 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:42.656933 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:42.656939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:42.656942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:42.657085 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:42.657824 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:42.657829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:42.658033 (+ 204us) client_negotiation.cc:769] Sending connection context
0807 02:55:42.658231 (+ 198us) client_negotiation.cc:240] Negotiation successful
0807 02:55:42.658597 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":578,"threads_started":1}
I20260807 02:55:42.661942 4177 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:38347
I20260807 02:55:42.661958 4358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:38347 every 8 connection(s)
I20260807 02:55:42.663650 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:55:42.665666 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 4177
I20260807 02:55:42.665915 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:55:42.666541 1509 external_mini_cluster.cc:1428] Setting key cefe9c630805e9a59c1b1bdc1e940b23
I20260807 02:55:42.667287 4361 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33811
I20260807 02:55:42.667531 4361 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:42.668349 4361 heartbeater.cc:511] Master 127.1.121.126:33811 requested a full tablet report, sending...
I20260807 02:55:42.670905 4085 ts_manager.cc:194] Registered new tserver with Master: 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347)
I20260807 02:55:42.672639 4085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:49955
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
W20260807 02:55:42.896256 1509 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.121.66: real 0.227s user 0.003s sys 0.012s
I20260807 02:55:42.897156 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33811
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:43.257470 4367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:43.257774 4367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:43.257889 4367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:43.266003 4367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:43.266151 4367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:43.266259 4367 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 02:55:43.266319 4367 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:55:43.266433 4367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:55:43.277455 4367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.121.126:33811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:43.278848 4367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:43.280229 4367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:43.295135 4372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:43.295172 4373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:43.295210 4375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:43.296001 4367 server_base.cc:1034] running on GCE node
I20260807 02:55:43.297806 4367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:43.303200 4367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:43.306658 4367 hybrid_clock.cc:648] HybridClock initialized: now 1786071343306538 us; error 129 us; skew 500 ppm
I20260807 02:55:43.307180 4367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:43 dist-test-slave-3h5h krb5kdc[4017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:43.311739 4367 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:55:43.313719 4367 webserver.cc:460] Webserver started at http://127.1.121.66:40101/ using document root <none> and password file <none>
I20260807 02:55:43.314554 4367 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:43.314715 4367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:43.315100 4367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:43.318620 4367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44"
format_stamp: "Formatted at 2026-08-07 02:55:43 on dist-test-slave-3h5h"
server_key: "dcd5e94913d0a7073bab8b161c5aac90"
server_key_iv: "e08132309e62585bdc518870d4e60a53"
server_key_version: "encryptionkey@0"
I20260807 02:55:43.319473 4367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44"
format_stamp: "Formatted at 2026-08-07 02:55:43 on dist-test-slave-3h5h"
server_key: "dcd5e94913d0a7073bab8b161c5aac90"
server_key_iv: "e08132309e62585bdc518870d4e60a53"
server_key_version: "encryptionkey@0"
I20260807 02:55:43.325407 4367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 02:55:43.329286 4387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:43.330785 4367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:55:43.330987 4367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44"
format_stamp: "Formatted at 2026-08-07 02:55:43 on dist-test-slave-3h5h"
server_key: "dcd5e94913d0a7073bab8b161c5aac90"
server_key_iv: "e08132309e62585bdc518870d4e60a53"
server_key_version: "encryptionkey@0"
I20260807 02:55:43.331219 4367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:43.361279 4367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:43.366400 4367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:43.366743 4367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:43.369935 4367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:43.370108 4367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:43.370328 4367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:43.370436 4367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:55:43 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:43.396240 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:43.371001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33811 (local address 127.1.121.66:50713)
0807 02:55:43.371919 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:43.371978 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:43.373951 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:43.374726 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:43.374734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:43.375261 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:43.376114 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:43.376153 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.378167 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.378178 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:43.380304 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:43.380316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.381428 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.381438 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:43.381647 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:43.382415 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:43.382436 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:43.386367 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:43.390489 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:43.390498 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:43.390514 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:43.391348 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:43.391972 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:43.391977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:43.391981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:43.392180 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:43.393092 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:43.393103 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:43.393539 (+ 436us) client_negotiation.cc:769] Sending connection context
0807 02:55:43.393970 (+ 431us) client_negotiation.cc:240] Negotiation successful
0807 02:55:43.394536 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":363,"threads_started":1}
I20260807 02:55:43.397424 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:43.370387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:50713 (local address 127.1.121.126:33811)
0807 02:55:43.370938 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:43.370988 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:43.372131 (+ 1143us) server_negotiation.cc:407] Connection header received
0807 02:55:43.374278 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:43.374284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:43.374375 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:43.374497 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:43.376356 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.377934 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.380457 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.381101 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.381255 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:43.386668 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:43.386677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:43.386683 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:43.386730 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:43.390340 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:43.391515 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:43.391593 (+ 78us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:43.391601 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:43.391817 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:43.392319 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:43.392328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:43.392335 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:43.392901 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:43.393091 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:43.396876 (+ 3785us) server_negotiation.cc:299] Negotiation successful
0807 02:55:43.397152 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":261,"threads_started":1}
I20260807 02:55:43.422976 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:43.404857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33811 (local address 127.1.121.66:36579)
0807 02:55:43.406536 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:43.406556 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:43.406742 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:43.407308 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:43.407311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:43.408194 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:43.408879 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:43.408898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.410297 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.410306 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:43.412135 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:43.412155 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.413301 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.413305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:43.413551 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:43.414331 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:43.414348 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:43.415325 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:43.419356 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:43.419361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:43.419365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:43.420017 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:43.420509 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:43.420517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:43.420537 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:43.420796 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:43.421835 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:43.421841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:43.422052 (+ 211us) client_negotiation.cc:769] Sending connection context
0807 02:55:43.422169 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 02:55:43.422453 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"thread_start_us":1365,"threads_started":1}
I20260807 02:55:43.423209 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:43.404793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:36579 (local address 127.1.121.126:33811)
0807 02:55:43.405083 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:43.405092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:43.406666 (+ 1574us) server_negotiation.cc:407] Connection header received
0807 02:55:43.406900 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:43.406908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:43.407019 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:43.407119 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:43.409039 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.410138 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.412353 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.412914 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.413065 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:43.415606 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:43.415613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:43.415618 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:43.415660 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:43.419215 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:43.420162 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:43.420167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:43.420171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:43.420353 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:43.421061 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:43.421067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:43.421070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:43.421682 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:43.421871 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:43.422350 (+ 479us) server_negotiation.cc:299] Negotiation successful
0807 02:55:43.422758 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260807 02:55:43.424396 4367 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:44555
I20260807 02:55:43.424491 4548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:44555 every 8 connection(s)
I20260807 02:55:43.425912 4367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:55:43.428292 4550 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33811
I20260807 02:55:43.428484 4550 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:43.429137 4550 heartbeater.cc:511] Master 127.1.121.126:33811 requested a full tablet report, sending...
I20260807 02:55:43.430480 4085 ts_manager.cc:194] Registered new tserver with Master: 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555)
I20260807 02:55:43.431330 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 4367
I20260807 02:55:43.431507 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:55:43.431551 4085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:50713
I20260807 02:55:43.432214 1509 external_mini_cluster.cc:1428] Setting key f6ffc36339fa8d2d1181a13c367086ba
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:55:43.484120 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33811
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:55:43.676339 4361 heartbeater.cc:503] Master 127.1.121.126:33811 was elected leader, sending a full tablet report...
W20260807 02:55:43.842028 4556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:43.842315 4556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:43.842528 4556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:43.850059 4556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:43.850196 4556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:43.850360 4556 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 02:55:43.850442 4556 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:55:43.850571 4556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:55:43.860669 4556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.121.126:33811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:43.862133 4556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:43.863518 4556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:43.877214 4561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:43.877292 4564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:43.877328 4562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:43.878728 4556 server_base.cc:1034] running on GCE node
I20260807 02:55:43.879527 4556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:43.880630 4556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:43.881879 4556 hybrid_clock.cc:648] HybridClock initialized: now 1786071343881825 us; error 41 us; skew 500 ppm
I20260807 02:55:43.882364 4556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:43 dist-test-slave-3h5h krb5kdc[4017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:43.886307 4556 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:55:43.888137 4556 webserver.cc:460] Webserver started at http://127.1.121.67:44515/ using document root <none> and password file <none>
I20260807 02:55:43.889002 4556 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:43.889171 4556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:43.889603 4556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:43.893293 4556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "78f127347f7049f5b7f1be70b2811f82"
format_stamp: "Formatted at 2026-08-07 02:55:43 on dist-test-slave-3h5h"
server_key: "ce2c9d5e6529ca34dee386bc9e5ad03b"
server_key_iv: "6400ac906dbc585be06e70cbeea2b1be"
server_key_version: "encryptionkey@0"
I20260807 02:55:43.894258 4556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "78f127347f7049f5b7f1be70b2811f82"
format_stamp: "Formatted at 2026-08-07 02:55:43 on dist-test-slave-3h5h"
server_key: "ce2c9d5e6529ca34dee386bc9e5ad03b"
server_key_iv: "6400ac906dbc585be06e70cbeea2b1be"
server_key_version: "encryptionkey@0"
I20260807 02:55:43.901401 4556 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260807 02:55:43.905328 4573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:43.907124 4556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:55:43.907341 4556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "78f127347f7049f5b7f1be70b2811f82"
format_stamp: "Formatted at 2026-08-07 02:55:43 on dist-test-slave-3h5h"
server_key: "ce2c9d5e6529ca34dee386bc9e5ad03b"
server_key_iv: "6400ac906dbc585be06e70cbeea2b1be"
server_key_version: "encryptionkey@0"
I20260807 02:55:43.907577 4556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:43.940054 4556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:43.944761 4556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:43.945176 4556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:43.948313 4556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:43.948530 4556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:43.948681 4556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:43.948786 4556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:55:43 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071343, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:43.973430 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:43.948611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:53109 (local address 127.1.121.126:33811)
0807 02:55:43.949165 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:43.949172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:43.950496 (+ 1324us) server_negotiation.cc:407] Connection header received
0807 02:55:43.952577 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:43.952583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:43.952683 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:43.952809 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:43.955375 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.956582 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.958403 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.958853 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.959001 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:43.964204 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:43.964209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:43.964212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:43.964248 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:43.968805 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:43.970149 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:43.970190 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:43.970193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:43.970422 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:43.971212 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:43.971219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:43.971225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:43.971627 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:43.971805 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:43.972710 (+ 905us) server_negotiation.cc:299] Negotiation successful
0807 02:55:43.973002 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
I20260807 02:55:43.974541 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:43.949331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33811 (local address 127.1.121.67:53109)
0807 02:55:43.950301 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:43.950360 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:43.952279 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:43.953090 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:43.953098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:43.953959 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:43.955138 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:43.955175 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.956750 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.956756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:43.958237 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:43.958251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.959118 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.959127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:43.959364 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:43.960110 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:43.960139 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:43.963925 (+ 3786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:43.969042 (+ 5117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:43.969056 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:43.969087 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:43.969939 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:43.970594 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:43.970603 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:43.970610 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:43.971027 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:43.971780 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:43.971789 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:43.972322 (+ 533us) client_negotiation.cc:769] Sending connection context
0807 02:55:43.972610 (+ 288us) client_negotiation.cc:240] Negotiation successful
0807 02:55:43.973297 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":411,"threads_started":1}
I20260807 02:55:44.002820 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:43.981561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33811 (local address 127.1.121.67:58827)
0807 02:55:43.985436 (+ 3875us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:43.985452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:43.985641 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:43.986220 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:43.986225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:43.987102 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:43.987618 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:43.987634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.989566 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.989572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:43.991327 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:43.991340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.992172 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.992178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:43.992344 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:43.993045 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:43.993066 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:43.994389 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:43.998456 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:43.998466 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:43.998473 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:43.999361 (+ 888us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:43.999863 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:43.999872 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:43.999879 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:44.000109 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:44.000912 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:44.000925 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:44.001254 (+ 329us) client_negotiation.cc:769] Sending connection context
0807 02:55:44.001534 (+ 280us) client_negotiation.cc:240] Negotiation successful
0807 02:55:44.002073 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3744,"thread_start_us":3595,"threads_started":1}
I20260807 02:55:44.003314 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:43.981918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:58827 (local address 127.1.121.126:33811)
0807 02:55:43.982165 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:43.982174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:43.985561 (+ 3387us) server_negotiation.cc:407] Connection header received
0807 02:55:43.985780 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:43.985786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:43.985883 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:43.986003 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:43.987852 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.989322 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.991491 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:43.991973 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:43.992123 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:43.994671 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:43.994678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:43.994689 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:43.994723 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:43.998228 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:43.999535 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:43.999541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:43.999546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:43.999717 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:44.000305 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:44.000310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:44.000314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:44.000756 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:44.000946 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.002675 (+ 1729us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.002957 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 02:55:44.009001 4556 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:39003
I20260807 02:55:44.009047 4741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:39003 every 8 connection(s)
I20260807 02:55:44.010519 4556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:55:44.012881 4742 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33811
I20260807 02:55:44.013180 4742 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:44.013970 4742 heartbeater.cc:511] Master 127.1.121.126:33811 requested a full tablet report, sending...
I20260807 02:55:44.015462 4085 ts_manager.cc:194] Registered new tserver with Master: 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:44.016266 4085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:53109
I20260807 02:55:44.018308 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 4556
I20260807 02:55:44.018486 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:55:44.019006 1509 external_mini_cluster.cc:1428] Setting key e406b7744f03e01ef4c9ac96b470fa11
I20260807 02:55:44.022194 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:55:44 dist-test-slave-3h5h krb5kdc[4017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:55:44 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:44.080788 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.065948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.1.121.126:33811)
0807 02:55:44.066195 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.066204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.066350 (+ 146us) server_negotiation.cc:407] Connection header received
0807 02:55:44.066602 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.066607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.066682 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.066799 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:44.068292 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.069205 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.070663 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.071183 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.071290 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.075266 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:44.075271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:44.075275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:44.075312 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:44.078197 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:44.078842 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:44.078847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:44.078850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:44.079013 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:44.079374 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:44.079377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:44.079379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:44.079701 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:44.079921 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.080147 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.080328 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 02:55:44.085950 4085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:55:44.098209 4085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:55:44.138670 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.112513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.1.121.65:38347)
0807 02:55:44.127354 (+ 14841us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.127370 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.127411 (+ 41us) server_negotiation.cc:407] Connection header received
0807 02:55:44.127546 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.127560 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.127921 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.129326 (+ 1405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:44.129546 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.132341 (+ 2795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.135523 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.137390 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.137535 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.137777 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.138142 (+ 365us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.138326 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14597,"thread_start_us":15591,"threads_started":1}
I20260807 02:55:44.138836 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.112235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38347 (local address 127.0.0.1:51140)
0807 02:55:44.123320 (+ 11085us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:44.123372 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:44.123581 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:44.128476 (+ 4895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:44.128485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:44.128522 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:44.129068 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.129092 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.132502 (+ 3410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.132508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.135344 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.135360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.137586 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.137592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.137895 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.137937 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 02:55:44.138076 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 02:55:44.138337 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10813,"mutex_wait_us":24,"thread_start_us":16909,"threads_started":1}
I20260807 02:55:44.140250 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.112247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:39003 (local address 127.0.0.1:43790)
0807 02:55:44.129451 (+ 17204us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:44.129472 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:44.129584 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:44.130902 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:44.130907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:44.130931 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:44.131671 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.131687 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.134529 (+ 2842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.134536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.137103 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.137120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.139388 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.139395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.139573 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.139615 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 02:55:44.139688 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 02:55:44.139834 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17070,"mutex_wait_us":56,"thread_start_us":19679,"threads_started":1}
I20260807 02:55:44.141149 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.112827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.1.121.67:39003)
0807 02:55:44.130295 (+ 17468us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.130303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.130336 (+ 33us) server_negotiation.cc:407] Connection header received
0807 02:55:44.130427 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.130434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.130692 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.131127 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:44.132796 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.134395 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.137318 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.139130 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.140115 (+ 985us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.140492 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.140576 (+ 84us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.140758 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17226,"thread_start_us":17021,"threads_started":1}
I20260807 02:55:44.141914 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.112751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:44555 (local address 127.0.0.1:48660)
0807 02:55:44.132325 (+ 19574us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:44.132345 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:44.132476 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:44.133258 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:44.133262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:44.133290 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:44.133761 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.133774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.135813 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.135818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.138644 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.138661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.140991 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.140996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.141129 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.141151 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 02:55:44.141270 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 02:55:44.141423 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19399,"mutex_wait_us":51,"thread_start_us":10117,"threads_started":1}
I20260807 02:55:44.142462 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.113330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.1.121.66:44555)
0807 02:55:44.128779 (+ 15449us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.128787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.132484 (+ 3697us) server_negotiation.cc:407] Connection header received
0807 02:55:44.132717 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.132738 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.133029 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.133258 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:44.133984 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.135594 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.138867 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.140415 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.140569 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.140752 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.141404 (+ 652us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.141593 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15263,"thread_start_us":15464,"threads_started":1}
I20260807 02:55:44.143189 4265 tablet_service.cc:1467] Processing CreateTablet for tablet d0008d022ac04e96977a9b348b137689 (DEFAULT_TABLE table=test-table [id=4daa965b1e4140ea8e2d896813084e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:44.144873 4265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0008d022ac04e96977a9b348b137689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:44.145414 4642 tablet_service.cc:1467] Processing CreateTablet for tablet d0008d022ac04e96977a9b348b137689 (DEFAULT_TABLE table=test-table [id=4daa965b1e4140ea8e2d896813084e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:44.147049 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0008d022ac04e96977a9b348b137689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:44.153033 4455 tablet_service.cc:1467] Processing CreateTablet for tablet d0008d022ac04e96977a9b348b137689 (DEFAULT_TABLE table=test-table [id=4daa965b1e4140ea8e2d896813084e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:44.154434 4455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0008d022ac04e96977a9b348b137689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:44.161151 4770 tablet_bootstrap.cc:492] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: Bootstrap starting.
I20260807 02:55:44.163419 4771 tablet_bootstrap.cc:492] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: Bootstrap starting.
I20260807 02:55:44.165617 4770 tablet_bootstrap.cc:654] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:44.166605 4771 tablet_bootstrap.cc:654] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:44.167220 4770 log.cc:824] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:44.167651 4771 log.cc:824] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:44.169601 4775 tablet_bootstrap.cc:492] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Bootstrap starting.
I20260807 02:55:44.170845 4770 tablet_bootstrap.cc:492] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: No bootstrap required, opened a new log
I20260807 02:55:44.171164 4770 ts_tablet_manager.cc:1397] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 02:55:44.172179 4771 tablet_bootstrap.cc:492] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: No bootstrap required, opened a new log
I20260807 02:55:44.172497 4771 ts_tablet_manager.cc:1397] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260807 02:55:44.173594 4775 tablet_bootstrap.cc:654] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:44.174923 4775 log.cc:824] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:44.178438 4775 tablet_bootstrap.cc:492] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: No bootstrap required, opened a new log
I20260807 02:55:44.178742 4775 ts_tablet_manager.cc:1397] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 02:55:44.181864 4771 raft_consensus.cc:348] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } }
I20260807 02:55:44.181905 4770 raft_consensus.cc:348] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } }
I20260807 02:55:44.182363 4770 raft_consensus.cc:374] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:44.182369 4771 raft_consensus.cc:374] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:44.182473 4771 raft_consensus.cc:729] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f127347f7049f5b7f1be70b2811f82, State: Initialized, Role: FOLLOWER
I20260807 02:55:44.182513 4770 raft_consensus.cc:729] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1d0d70d38444e7b8729d86e1445763, State: Initialized, Role: FOLLOWER
I20260807 02:55:44.183225 4770 consensus_queue.cc:260] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } }
I20260807 02:55:44.183401 4771 consensus_queue.cc:260] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } }
I20260807 02:55:44.186570 4742 heartbeater.cc:503] Master 127.1.121.126:33811 was elected leader, sending a full tablet report...
I20260807 02:55:44.187415 4771 ts_tablet_manager.cc:1428] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260807 02:55:44.189307 4770 ts_tablet_manager.cc:1428] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260807 02:55:44.191993 4775 raft_consensus.cc:348] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } }
I20260807 02:55:44.192430 4775 raft_consensus.cc:374] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:44.192577 4775 raft_consensus.cc:729] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9896e83f5a264d30b3dbcf0cf3c4ad44, State: Initialized, Role: FOLLOWER
I20260807 02:55:44.193341 4775 consensus_queue.cc:260] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } }
I20260807 02:55:44.194857 4550 heartbeater.cc:503] Master 127.1.121.126:33811 was elected leader, sending a full tablet report...
I20260807 02:55:44.195281 4775 ts_tablet_manager.cc:1428] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260807 02:55:44.262427 4743 tablet.cc:2369] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:55:44.400444 4777 raft_consensus.cc:482] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:44.400888 4777 raft_consensus.cc:504] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } }
I20260807 02:55:44.402801 4777 leader_election.cc:288] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003), 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555)
I20260807 02:55:44.410135 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.403469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:58513 (local address 127.1.121.66:44555)
0807 02:55:44.403761 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.403771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.404656 (+ 885us) server_negotiation.cc:407] Connection header received
0807 02:55:44.404905 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.404910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.404969 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.405072 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:44.405741 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.406498 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.408376 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.409200 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.409397 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.409473 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.409733 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.409860 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:55:44.410283 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.403455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:44555 (local address 127.1.121.65:58513)
0807 02:55:44.404548 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:44.404568 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:44.404714 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:44.405087 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:44.405092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:44.405130 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:44.405564 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.405578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.406628 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.406634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.408220 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.408232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.409405 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.409411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.409567 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.409591 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:55:44.409708 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 02:55:44.409838 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":24,"spinlock_wait_cycles":3328,"thread_start_us":714,"threads_started":1}
I20260807 02:55:44.410672 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.403200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:60317 (local address 127.1.121.67:39003)
0807 02:55:44.403432 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.403440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.404558 (+ 1118us) server_negotiation.cc:407] Connection header received
0807 02:55:44.404815 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.404820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.404949 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.405072 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:44.405859 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.406668 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.408683 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.409614 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.409839 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.409923 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.410237 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.410377 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 02:55:44.410948 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.403259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:39003 (local address 127.1.121.65:60317)
0807 02:55:44.404382 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:44.404415 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:44.404642 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:44.405091 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:44.405096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:44.405146 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:44.405693 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.405703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.406838 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.406846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.408512 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.408526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.409869 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.409876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.410055 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.410081 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 02:55:44.410208 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 02:55:44.410350 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":39,"spinlock_wait_cycles":1408,"thread_start_us":635,"threads_started":1}
I20260807 02:55:44.411688 4490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0008d022ac04e96977a9b348b137689" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" is_pre_election: true
I20260807 02:55:44.411921 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0008d022ac04e96977a9b348b137689" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82" is_pre_election: true
I20260807 02:55:44.412304 4490 raft_consensus.cc:2454] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 0.
I20260807 02:55:44.412429 4679 raft_consensus.cc:2454] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 0.
I20260807 02:55:44.413100 4201 leader_election.cc:302] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f127347f7049f5b7f1be70b2811f82, 9d1d0d70d38444e7b8729d86e1445763; no voters:
I20260807 02:55:44.413477 4777 raft_consensus.cc:2781] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:55:44.413636 4777 raft_consensus.cc:482] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:55:44.413769 4777 raft_consensus.cc:3037] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:44.415951 4777 raft_consensus.cc:504] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } }
W20260807 02:55:44.416692 4363 tablet.cc:2369] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:55:44.416749 4777 leader_election.cc:288] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 election: Requested vote from peers 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003), 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555)
I20260807 02:55:44.417104 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0008d022ac04e96977a9b348b137689" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82"
I20260807 02:55:44.417316 4679 raft_consensus.cc:3037] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:44.417652 4490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0008d022ac04e96977a9b348b137689" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44"
I20260807 02:55:44.417876 4490 raft_consensus.cc:3037] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:44.420650 4679 raft_consensus.cc:2454] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 1.
I20260807 02:55:44.421141 4201 leader_election.cc:302] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f127347f7049f5b7f1be70b2811f82, 9d1d0d70d38444e7b8729d86e1445763; no voters:
I20260807 02:55:44.421278 4490 raft_consensus.cc:2454] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 1.
I20260807 02:55:44.421473 4777 raft_consensus.cc:2781] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:44.421977 4777 raft_consensus.cc:686] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1d0d70d38444e7b8729d86e1445763, State: Running, Role: LEADER
I20260807 02:55:44.422523 4777 consensus_queue.cc:237] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } }
I20260807 02:55:44.427261 4085 catalog_manager.cc:5495] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65). New cstate: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } health_report { overall_health: UNKNOWN } } }
W20260807 02:55:44.428761 4551 tablet.cc:2369] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 02:55:44 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:44.542619 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.527661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33811 (local address 127.0.0.1:46316)
0807 02:55:44.528361 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:44.528377 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:44.528569 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:44.528929 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:44.528933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:44.529985 (+ 1052us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:44.530375 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.530394 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.531741 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.531748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.533086 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.533100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.533883 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.533888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.534075 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.534771 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:44.534789 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:44.537049 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:44.540260 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:44.540270 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:44.540276 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:44.540710 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:44.541152 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:44.541155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:44.541157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:44.541316 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:44.541821 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:44.541827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:44.541980 (+ 153us) client_negotiation.cc:769] Sending connection context
0807 02:55:44.542133 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 02:55:44.542328 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":424,"threads_started":1}
I20260807 02:55:44.542928 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.527633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.1.121.126:33811)
0807 02:55:44.527899 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.527908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.528488 (+ 580us) server_negotiation.cc:407] Connection header received
0807 02:55:44.528702 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.528711 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.528764 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.528906 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:44.530587 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.531581 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.533232 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.533678 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.533838 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.537213 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:44.537219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:44.537222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:44.537252 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:44.540126 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:44.540829 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:44.540886 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:44.540892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:44.541046 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:44.541417 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:44.541420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:44.541422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:44.541708 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:44.541828 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.542239 (+ 411us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.542467 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":84}
I20260807 02:55:44.548274 4085 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:46316:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 02:55:44.549362 4085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:55:44.555944 4265 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2dfd6f878645979c1c4ece440458f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0268d00362cd487bb6376eee884c1ed1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:55:44.556249 4455 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2dfd6f878645979c1c4ece440458f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0268d00362cd487bb6376eee884c1ed1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:55:44.557027 4265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2dfd6f878645979c1c4ece440458f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:44.557022 4455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2dfd6f878645979c1c4ece440458f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:44.556535 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2dfd6f878645979c1c4ece440458f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0268d00362cd487bb6376eee884c1ed1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:55:44.557626 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2dfd6f878645979c1c4ece440458f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:44.563869 4771 tablet_bootstrap.cc:492] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: Bootstrap starting.
I20260807 02:55:44.564283 4770 tablet_bootstrap.cc:492] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: Bootstrap starting.
I20260807 02:55:44.564704 4775 tablet_bootstrap.cc:492] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Bootstrap starting.
I20260807 02:55:44.565804 4770 tablet_bootstrap.cc:654] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:44.566036 4771 tablet_bootstrap.cc:654] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:44.566856 4775 tablet_bootstrap.cc:654] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:44.567916 4770 tablet_bootstrap.cc:492] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: No bootstrap required, opened a new log
I20260807 02:55:44.568053 4770 ts_tablet_manager.cc:1397] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 02:55:44.568712 4770 raft_consensus.cc:348] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.568979 4770 raft_consensus.cc:374] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:44.569149 4770 raft_consensus.cc:729] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1d0d70d38444e7b8729d86e1445763, State: Initialized, Role: FOLLOWER
I20260807 02:55:44.569401 4770 consensus_queue.cc:260] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.570115 4770 ts_tablet_manager.cc:1428] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 02:55:44.569957 4777 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.570300 4777 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:44.572117 4775 tablet_bootstrap.cc:492] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: No bootstrap required, opened a new log
I20260807 02:55:44.572278 4775 ts_tablet_manager.cc:1397] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260807 02:55:44.572392 4771 tablet_bootstrap.cc:492] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: No bootstrap required, opened a new log
I20260807 02:55:44.572542 4771 ts_tablet_manager.cc:1397] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260807 02:55:44.573120 4775 raft_consensus.cc:348] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.573386 4775 raft_consensus.cc:374] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:44.573473 4775 raft_consensus.cc:729] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9896e83f5a264d30b3dbcf0cf3c4ad44, State: Initialized, Role: FOLLOWER
I20260807 02:55:44.573318 4771 raft_consensus.cc:348] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.573573 4771 raft_consensus.cc:374] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:44.573660 4771 raft_consensus.cc:729] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f127347f7049f5b7f1be70b2811f82, State: Initialized, Role: FOLLOWER
I20260807 02:55:44.573726 4775 consensus_queue.cc:260] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.574395 4775 ts_tablet_manager.cc:1428] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 02:55:44.574246 4779 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.574646 4779 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:44.573948 4771 consensus_queue.cc:260] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.575953 4771 ts_tablet_manager.cc:1428] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:55:44.575806 4778 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.576208 4778 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:44.738162 4777 raft_consensus.cc:482] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:44.738384 4777 raft_consensus.cc:504] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.739105 4777 leader_election.cc:288] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555), 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:44.739715 4490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2dfd6f878645979c1c4ece440458f8" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" is_pre_election: true
I20260807 02:55:44.739804 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2dfd6f878645979c1c4ece440458f8" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82" is_pre_election: true
I20260807 02:55:44.740041 4490 raft_consensus.cc:2454] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 0.
I20260807 02:55:44.740073 4679 raft_consensus.cc:2454] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 0.
I20260807 02:55:44.740548 4201 leader_election.cc:302] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f127347f7049f5b7f1be70b2811f82, 9d1d0d70d38444e7b8729d86e1445763; no voters:
I20260807 02:55:44.740871 4777 raft_consensus.cc:2781] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:55:44.740975 4777 raft_consensus.cc:482] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:55:44.741096 4777 raft_consensus.cc:3037] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:44.743117 4777 raft_consensus.cc:504] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.743726 4777 leader_election.cc:288] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 election: Requested vote from peers 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555), 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:44.744170 4490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2dfd6f878645979c1c4ece440458f8" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44"
I20260807 02:55:44.744215 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2dfd6f878645979c1c4ece440458f8" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82"
I20260807 02:55:44.744422 4490 raft_consensus.cc:3037] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:44.744448 4679 raft_consensus.cc:3037] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:44.746548 4679 raft_consensus.cc:2454] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 1.
I20260807 02:55:44.746766 4490 raft_consensus.cc:2454] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 1.
I20260807 02:55:44.747059 4201 leader_election.cc:302] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f127347f7049f5b7f1be70b2811f82, 9d1d0d70d38444e7b8729d86e1445763; no voters:
I20260807 02:55:44.747334 4777 raft_consensus.cc:2781] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:44.747570 4777 raft_consensus.cc:686] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1d0d70d38444e7b8729d86e1445763, State: Running, Role: LEADER
I20260807 02:55:44.747853 4777 consensus_queue.cc:237] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.749686 4786 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: TxnStatusTablet state changed. Reason: New leader 9d1d0d70d38444e7b8729d86e1445763. Latest consensus state: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.750059 4786 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:55:44.750725 4802 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 02:55:44.751046 4802 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 02:55:44.751286 4084 catalog_manager.cc:5495] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65). New cstate: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260807 02:55:44.773543 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.765757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.1.121.65:38347)
0807 02:55:44.766443 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.766448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.766471 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:55:44.766552 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.766557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.766645 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.766783 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:44.767318 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.768327 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.770673 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.771212 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.771317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.772087 (+ 770us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:44.772737 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:44.772954 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.773155 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.773225 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":406,"threads_started":1}
I20260807 02:55:44.773638 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.765432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38347 (local address 127.0.0.1:51154)
0807 02:55:44.765731 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:44.765754 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:44.765904 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:44.766851 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:44.766857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:44.766872 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:55:44.767097 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.767110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.768478 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.768485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.770520 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.770539 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.771404 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.771412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.771669 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.771752 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:55:44.772918 (+ 1166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:55:44.772951 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 02:55:44.773076 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 02:55:44.773253 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 02:55:44.782186 4679 raft_consensus.cc:1261] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Refusing update from remote peer 9d1d0d70d38444e7b8729d86e1445763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:44.782186 4490 raft_consensus.cc:1261] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Refusing update from remote peer 9d1d0d70d38444e7b8729d86e1445763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:44.783339 4786 consensus_queue.cc:1035] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:44.783793 4777 consensus_queue.cc:1035] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:44.791886 4779 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: TxnStatusTablet state changed. Reason: New leader 9d1d0d70d38444e7b8729d86e1445763. Latest consensus state: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.792207 4779 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:44.795878 4786 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.796190 4786 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:55:44.820464 4778 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: TxnStatusTablet state changed. Reason: New leader 9d1d0d70d38444e7b8729d86e1445763. Latest consensus state: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.820859 4778 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:44.822642 4779 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.822916 4779 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 9896e83f5a264d30b3dbcf0cf3c4ad44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:44.825273 4777 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.825615 4777 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 9d1d0d70d38444e7b8729d86e1445763: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:55:44.826238 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.814988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.1.121.65:38347)
0807 02:55:44.815244 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.815254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.818192 (+ 2938us) server_negotiation.cc:407] Connection header received
0807 02:55:44.818438 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.818445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.818531 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.818629 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:44.819373 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.820558 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.823744 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.824429 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.824554 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.825225 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:44.825367 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:44.825433 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.825859 (+ 426us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.825945 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:55:44.826635 4808 mvcc.cc:205] Tried to move back new op lower bound from 7315748228216496128 to 7315748228089704448. Current Snapshot: MvccSnapshot[applied={T|T < 7315748228216496128}]
I20260807 02:55:44.828990 4778 tablet_replica.cc:439] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } } }
I20260807 02:55:44.829221 4778 tablet_replica.cc:442] T 1c2dfd6f878645979c1c4ece440458f8 P 78f127347f7049f5b7f1be70b2811f82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:55:44.845949 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.839594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:56809 (local address 127.1.121.65:38347)
0807 02:55:44.839835 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:44.839844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:44.840286 (+ 442us) server_negotiation.cc:407] Connection header received
0807 02:55:44.840499 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:44.840505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:44.840602 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:44.840727 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:44.841346 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.842197 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.843845 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.844312 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.844462 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.845042 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:44.845148 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:44.845256 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:44.845410 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 02:55:44.845511 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260807 02:55:44.846097 4820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:44.839513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38347 (local address 127.1.121.65:56809)
0807 02:55:44.840184 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:44.840200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:44.840368 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:44.840746 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:44.840751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:44.840762 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:55:44.841185 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.841196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.842335 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.842344 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.843685 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:44.843699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:44.844545 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:44.844552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:44.844690 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:44.844742 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:55:44.845256 (+ 514us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:55:44.845271 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 02:55:44.845372 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 02:55:44.845572 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":46,"thread_start_us":363,"threads_started":1}
I20260807 02:55:44.856091 4490 raft_consensus.cc:1261] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Refusing update from remote peer 9d1d0d70d38444e7b8729d86e1445763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:44.856292 4679 raft_consensus.cc:1261] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Refusing update from remote peer 9d1d0d70d38444e7b8729d86e1445763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:44.856696 4786 consensus_queue.cc:1035] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:44.856989 4777 consensus_queue.cc:1035] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:44.862376 4806 mvcc.cc:205] Tried to move back new op lower bound from 7315748228525694976 to 7315748226758803456. Current Snapshot: MvccSnapshot[applied={T|T < 7315748228525694976}]
I20260807 02:55:44.929450 4084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:44.929760 4084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:44.933661 4084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71: Sending DeleteTablet for 3 replicas of tablet d0008d022ac04e96977a9b348b137689
I20260807 02:55:44.934846 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet d0008d022ac04e96977a9b348b137689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:44 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:43790
I20260807 02:55:44.934902 4455 tablet_service.cc:1514] Processing DeleteTablet for tablet d0008d022ac04e96977a9b348b137689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:44 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:48660
I20260807 02:55:44.934897 4264 tablet_service.cc:1514] Processing DeleteTablet for tablet d0008d022ac04e96977a9b348b137689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:44 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:51140
I20260807 02:55:44.938871 4084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:55:44.939783 4084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:55:44.945933 4833 tablet_replica.cc:330] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: stopping tablet replica
I20260807 02:55:44.947680 4833 raft_consensus.cc:2229] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:44.948120 4833 raft_consensus.cc:2258] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:44.948953 4832 tablet_replica.cc:330] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: stopping tablet replica
I20260807 02:55:44.949412 4832 raft_consensus.cc:2229] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:44.949777 4832 raft_consensus.cc:2258] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:44.950538 4265 tablet_service.cc:1467] Processing CreateTablet for tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d (DEFAULT_TABLE table=test-table [id=9191f290cb7346b9bb4b10785abe7473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:44.950676 4642 tablet_service.cc:1467] Processing CreateTablet for tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d (DEFAULT_TABLE table=test-table [id=9191f290cb7346b9bb4b10785abe7473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:44.951022 4265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:44.951073 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:44.952039 4832 ts_tablet_manager.cc:1905] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:44.956146 4455 tablet_service.cc:1467] Processing CreateTablet for tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d (DEFAULT_TABLE table=test-table [id=9191f290cb7346b9bb4b10785abe7473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:44.956617 4455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:44.958012 4771 tablet_bootstrap.cc:492] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: Bootstrap starting.
I20260807 02:55:44.960144 4832 ts_tablet_manager.cc:1918] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:55:44.960203 4771 tablet_bootstrap.cc:654] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:44.960404 4832 log.cc:1196] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/d0008d022ac04e96977a9b348b137689
I20260807 02:55:44.961073 4832 ts_tablet_manager.cc:1939] T d0008d022ac04e96977a9b348b137689 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting consensus metadata
I20260807 02:55:44.961158 4770 tablet_bootstrap.cc:492] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: Bootstrap starting.
I20260807 02:55:44.963167 4067 catalog_manager.cc:4841] TS 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555): tablet d0008d022ac04e96977a9b348b137689 (table test-table [id=4daa965b1e4140ea8e2d896813084e0a]) successfully deleted
I20260807 02:55:44.963539 4830 tablet_replica.cc:330] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: stopping tablet replica
I20260807 02:55:44.964020 4830 raft_consensus.cc:2229] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:55:44.965165 4833 ts_tablet_manager.cc:1905] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:44.965332 4770 tablet_bootstrap.cc:654] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:44.969049 4830 raft_consensus.cc:2258] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:44.969925 4775 tablet_bootstrap.cc:492] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: Bootstrap starting.
I20260807 02:55:44.971360 4771 tablet_bootstrap.cc:492] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: No bootstrap required, opened a new log
I20260807 02:55:44.971634 4771 ts_tablet_manager.cc:1397] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20260807 02:55:44.971681 4775 tablet_bootstrap.cc:654] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:44.971829 4830 ts_tablet_manager.cc:1905] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:44.973397 4833 ts_tablet_manager.cc:1918] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:55:44.973573 4833 log.cc:1196] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/d0008d022ac04e96977a9b348b137689
I20260807 02:55:44.974129 4833 ts_tablet_manager.cc:1939] T d0008d022ac04e96977a9b348b137689 P 78f127347f7049f5b7f1be70b2811f82: Deleting consensus metadata
I20260807 02:55:44.972908 4771 raft_consensus.cc:348] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.974980 4771 raft_consensus.cc:374] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:44.975157 4771 raft_consensus.cc:729] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f127347f7049f5b7f1be70b2811f82, State: Initialized, Role: FOLLOWER
I20260807 02:55:44.975903 4771 consensus_queue.cc:260] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.978222 4775 tablet_bootstrap.cc:492] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: No bootstrap required, opened a new log
I20260807 02:55:44.978435 4775 ts_tablet_manager.cc:1397] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 02:55:44.979306 4830 ts_tablet_manager.cc:1918] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:55:44.979240 4775 raft_consensus.cc:348] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.979485 4830 log.cc:1196] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/d0008d022ac04e96977a9b348b137689
I20260807 02:55:44.979650 4775 raft_consensus.cc:374] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:44.979774 4775 raft_consensus.cc:729] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9896e83f5a264d30b3dbcf0cf3c4ad44, State: Initialized, Role: FOLLOWER
I20260807 02:55:44.980170 4830 ts_tablet_manager.cc:1939] T d0008d022ac04e96977a9b348b137689 P 9d1d0d70d38444e7b8729d86e1445763: Deleting consensus metadata
I20260807 02:55:44.980103 4775 consensus_queue.cc:260] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.980424 4771 ts_tablet_manager.cc:1428] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260807 02:55:44.980715 4770 tablet_bootstrap.cc:492] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: No bootstrap required, opened a new log
I20260807 02:55:44.980909 4770 ts_tablet_manager.cc:1397] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20260807 02:55:44.981694 4775 ts_tablet_manager.cc:1428] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260807 02:55:44.981678 4770 raft_consensus.cc:348] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.981932 4770 raft_consensus.cc:374] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:44.982043 4770 raft_consensus.cc:729] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1d0d70d38444e7b8729d86e1445763, State: Initialized, Role: FOLLOWER
I20260807 02:55:44.982304 4770 consensus_queue.cc:260] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:44.983526 4770 ts_tablet_manager.cc:1428] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:55:44.983745 4066 catalog_manager.cc:4841] TS 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003): tablet d0008d022ac04e96977a9b348b137689 (table test-table [id=4daa965b1e4140ea8e2d896813084e0a]) successfully deleted
I20260807 02:55:44.985321 4064 catalog_manager.cc:4841] TS 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347): tablet d0008d022ac04e96977a9b348b137689 (table test-table [id=4daa965b1e4140ea8e2d896813084e0a]) successfully deleted
I20260807 02:55:45.087821 4779 raft_consensus.cc:482] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:45.088107 4779 raft_consensus.cc:504] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:45.089891 4779 leader_election.cc:288] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347), 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:45.105309 4827 raft_consensus.cc:482] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:45.105564 4827 raft_consensus.cc:504] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:45.106803 4827 leader_election.cc:288] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555), 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:45.107470 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01dd7ef5a5b4bda81ce7bf1f6ab576d" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82" is_pre_election: true
I20260807 02:55:45.107762 4679 raft_consensus.cc:2454] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 0.
I20260807 02:55:45.108198 4490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01dd7ef5a5b4bda81ce7bf1f6ab576d" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" is_pre_election: true
I20260807 02:55:45.108445 4490 raft_consensus.cc:2454] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 0.
I20260807 02:55:45.109110 4201 leader_election.cc:302] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f127347f7049f5b7f1be70b2811f82, 9d1d0d70d38444e7b8729d86e1445763; no voters:
I20260807 02:55:45.109436 4827 raft_consensus.cc:2781] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:55:45.109565 4827 raft_consensus.cc:482] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:55:45.109676 4827 raft_consensus.cc:3037] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:45.113122 4827 raft_consensus.cc:504] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:45.113454 4838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:45.090626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:39003 (local address 127.1.121.66:53227)
0807 02:55:45.103162 (+ 12536us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:45.103185 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:45.103333 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:45.103850 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:45.103862 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:45.103891 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:45.104398 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:45.104414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:45.107940 (+ 3526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:45.107947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:45.110241 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:45.110261 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:45.112372 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:45.112380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:45.112591 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:45.112631 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 02:55:45.112744 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 02:55:45.112915 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12399,"mutex_wait_us":1,"thread_start_us":12145,"threads_started":1}
I20260807 02:55:45.114449 4490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01dd7ef5a5b4bda81ce7bf1f6ab576d" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44"
I20260807 02:55:45.114548 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:45.090255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:36851 (local address 127.1.121.65:38347)
0807 02:55:45.090535 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:45.090543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:45.100656 (+ 10113us) server_negotiation.cc:407] Connection header received
0807 02:55:45.103084 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:45.103091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:45.103197 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:45.103261 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:45.103991 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:45.105148 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:45.107875 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:45.109301 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:45.109459 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:45.109541 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:45.114111 (+ 4570us) server_negotiation.cc:299] Negotiation successful
0807 02:55:45.114269 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260807 02:55:45.114564 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:45.091073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:53227 (local address 127.1.121.67:39003)
0807 02:55:45.102735 (+ 11662us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:45.102742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:45.103469 (+ 727us) server_negotiation.cc:407] Connection header received
0807 02:55:45.103567 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:45.103573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:45.103660 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:45.104590 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:55:45.104825 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:45.107729 (+ 2904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:45.110504 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:45.111880 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:45.112110 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:45.112199 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:45.114017 (+ 1818us) server_negotiation.cc:299] Negotiation successful
0807 02:55:45.114184 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11529,"thread_start_us":11437,"threads_started":1}
I20260807 02:55:45.114681 4490 raft_consensus.cc:3037] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:45.116000 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:45.090215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38347 (local address 127.1.121.66:36851)
0807 02:55:45.100509 (+ 10294us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:45.100536 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:45.101222 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:45.103476 (+ 2254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:45.103481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:45.103506 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:55:45.103806 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:45.103816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:45.105336 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:45.105341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:45.107599 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:45.107614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:45.109894 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:45.109901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:45.113902 (+ 4001us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:45.113943 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 02:55:45.115428 (+ 1485us) client_negotiation.cc:240] Negotiation successful
0807 02:55:45.115592 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10106,"mutex_wait_us":3491,"thread_start_us":9936,"threads_started":1}
I20260807 02:55:45.117297 4300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01dd7ef5a5b4bda81ce7bf1f6ab576d" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1d0d70d38444e7b8729d86e1445763" is_pre_election: true
I20260807 02:55:45.117933 4300 raft_consensus.cc:2379] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in current term 1: Already voted for candidate 9d1d0d70d38444e7b8729d86e1445763 in this term.
I20260807 02:55:45.117960 4490 raft_consensus.cc:2454] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 1.
I20260807 02:55:45.118405 4202 leader_election.cc:302] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9896e83f5a264d30b3dbcf0cf3c4ad44, 9d1d0d70d38444e7b8729d86e1445763; no voters:
I20260807 02:55:45.118764 4828 raft_consensus.cc:2781] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:45.118759 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01dd7ef5a5b4bda81ce7bf1f6ab576d" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82" is_pre_election: true
I20260807 02:55:45.118944 4828 raft_consensus.cc:686] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1d0d70d38444e7b8729d86e1445763, State: Running, Role: LEADER
I20260807 02:55:45.118988 4679 raft_consensus.cc:2454] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in term 0.
I20260807 02:55:45.119295 4828 consensus_queue.cc:237] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:45.119621 4389 leader_election.cc:302] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f127347f7049f5b7f1be70b2811f82, 9896e83f5a264d30b3dbcf0cf3c4ad44; no voters:
I20260807 02:55:45.120033 4779 raft_consensus.cc:2741] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260807 02:55:45.121351 4827 leader_election.cc:288] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [CANDIDATE]: Term 1 election: Requested vote from peers 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555), 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:45.121658 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01dd7ef5a5b4bda81ce7bf1f6ab576d" candidate_uuid: "9d1d0d70d38444e7b8729d86e1445763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82"
I20260807 02:55:45.121871 4679 raft_consensus.cc:3037] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:45.124753 4679 raft_consensus.cc:2454] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1d0d70d38444e7b8729d86e1445763 in term 1.
I20260807 02:55:45.124894 4082 catalog_manager.cc:5495] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65). New cstate: current_term: 1 leader_uuid: "9d1d0d70d38444e7b8729d86e1445763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260807 02:55:45.153393 4679 raft_consensus.cc:1261] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Refusing update from remote peer 9d1d0d70d38444e7b8729d86e1445763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:45.153436 4488 raft_consensus.cc:1261] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Refusing update from remote peer 9d1d0d70d38444e7b8729d86e1445763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:45.154107 4828 consensus_queue.cc:1035] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:45.154940 4827 consensus_queue.cc:1035] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 02:55:45.181942 4806 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 02:55:46.799109 4082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:46.799401 4082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:46.805004 4082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71: Sending DeleteTablet for 3 replicas of tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d
I20260807 02:55:46.806064 4455 tablet_service.cc:1514] Processing DeleteTablet for tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:46 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:48660
I20260807 02:55:46.806470 4264 tablet_service.cc:1514] Processing DeleteTablet for tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:46 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:51140
I20260807 02:55:46.806864 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:46 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:43790
I20260807 02:55:46.806936 4877 tablet_replica.cc:330] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: stopping tablet replica
I20260807 02:55:46.807296 4877 raft_consensus.cc:2229] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:46.807775 4877 raft_consensus.cc:2258] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:46.809391 4877 ts_tablet_manager.cc:1905] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:46.819502 4879 tablet_replica.cc:330] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: stopping tablet replica
I20260807 02:55:46.819808 4877 ts_tablet_manager.cc:1918] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:55:46.819968 4879 raft_consensus.cc:2229] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:46.820180 4877 log.cc:1196] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/a01dd7ef5a5b4bda81ce7bf1f6ab576d
I20260807 02:55:46.820626 4879 raft_consensus.cc:2258] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:46.821933 4878 tablet_replica.cc:330] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: stopping tablet replica
I20260807 02:55:46.822412 4878 raft_consensus.cc:2229] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:55:46.822690 4879 ts_tablet_manager.cc:1905] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:46.821534 4877 ts_tablet_manager.cc:1939] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting consensus metadata
I20260807 02:55:46.822969 4878 raft_consensus.cc:2258] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:46.824062 4878 ts_tablet_manager.cc:1905] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:46.824239 4067 catalog_manager.cc:4841] TS 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555): tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d (table test-table [id=9191f290cb7346b9bb4b10785abe7473]) successfully deleted
I20260807 02:55:46.830276 4879 ts_tablet_manager.cc:1918] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:55:46.830474 4879 log.cc:1196] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/a01dd7ef5a5b4bda81ce7bf1f6ab576d
I20260807 02:55:46.831027 4879 ts_tablet_manager.cc:1939] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 78f127347f7049f5b7f1be70b2811f82: Deleting consensus metadata
I20260807 02:55:46.831903 4878 ts_tablet_manager.cc:1918] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:55:46.832194 4878 log.cc:1196] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/a01dd7ef5a5b4bda81ce7bf1f6ab576d
I20260807 02:55:46.832582 4066 catalog_manager.cc:4841] TS 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003): tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d (table test-table [id=9191f290cb7346b9bb4b10785abe7473]) successfully deleted
I20260807 02:55:46.833019 4878 ts_tablet_manager.cc:1939] T a01dd7ef5a5b4bda81ce7bf1f6ab576d P 9d1d0d70d38444e7b8729d86e1445763: Deleting consensus metadata
I20260807 02:55:46.834205 4064 catalog_manager.cc:4841] TS 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347): tablet a01dd7ef5a5b4bda81ce7bf1f6ab576d (table test-table [id=9191f290cb7346b9bb4b10785abe7473]) successfully deleted
Aug 07 02:55:46 dist-test-slave-3h5h krb5kdc[4017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071346, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:55:46 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071346, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:46.878574 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:46.858139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.1.121.126:33811)
0807 02:55:46.859258 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:46.859262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:46.859284 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:55:46.859342 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:46.859345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:46.859405 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:46.859500 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:46.861293 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:46.862122 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:46.863686 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:46.864095 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:46.864195 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:46.872726 (+ 8531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:46.872732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:46.872735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:46.872771 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:46.875661 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:46.876482 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:46.876486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:46.876488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:46.876648 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:46.877040 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:46.877044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:46.877046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:46.877360 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:46.877555 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:46.877919 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 02:55:46.878166 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":789,"threads_started":1}
I20260807 02:55:46.882447 4082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:55:46.883023 4082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:55:46.890025 4264 tablet_service.cc:1467] Processing CreateTablet for tablet 84392126894c45d1a91032e79d417ec6 (DEFAULT_TABLE table=test-table [id=e21b35b94ec34f34b96d118809ba70c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:46.890142 4455 tablet_service.cc:1467] Processing CreateTablet for tablet 84392126894c45d1a91032e79d417ec6 (DEFAULT_TABLE table=test-table [id=e21b35b94ec34f34b96d118809ba70c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:46.890592 4455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84392126894c45d1a91032e79d417ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:46.890468 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 84392126894c45d1a91032e79d417ec6 (DEFAULT_TABLE table=test-table [id=e21b35b94ec34f34b96d118809ba70c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:46.890618 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84392126894c45d1a91032e79d417ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:46.890904 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84392126894c45d1a91032e79d417ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:46.896883 4892 tablet_bootstrap.cc:492] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Bootstrap starting.
I20260807 02:55:46.898964 4894 tablet_bootstrap.cc:492] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: Bootstrap starting.
I20260807 02:55:46.900038 4893 tablet_bootstrap.cc:492] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: Bootstrap starting.
I20260807 02:55:46.900117 4892 tablet_bootstrap.cc:654] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:46.901006 4894 tablet_bootstrap.cc:654] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:46.902029 4893 tablet_bootstrap.cc:654] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:46.904071 4894 tablet_bootstrap.cc:492] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: No bootstrap required, opened a new log
I20260807 02:55:46.904263 4894 ts_tablet_manager.cc:1397] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260807 02:55:46.905453 4894 raft_consensus.cc:348] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:46.905802 4894 raft_consensus.cc:374] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:46.905983 4894 raft_consensus.cc:729] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f127347f7049f5b7f1be70b2811f82, State: Initialized, Role: FOLLOWER
I20260807 02:55:46.906314 4894 consensus_queue.cc:260] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:46.908005 4894 ts_tablet_manager.cc:1428] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260807 02:55:46.908427 4892 tablet_bootstrap.cc:492] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: No bootstrap required, opened a new log
I20260807 02:55:46.908613 4892 ts_tablet_manager.cc:1397] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260807 02:55:46.909305 4893 tablet_bootstrap.cc:492] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: No bootstrap required, opened a new log
I20260807 02:55:46.909458 4893 ts_tablet_manager.cc:1397] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260807 02:55:46.909528 4892 raft_consensus.cc:348] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:46.909909 4892 raft_consensus.cc:374] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:46.910061 4892 raft_consensus.cc:729] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9896e83f5a264d30b3dbcf0cf3c4ad44, State: Initialized, Role: FOLLOWER
I20260807 02:55:46.910369 4892 consensus_queue.cc:260] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:46.910594 4893 raft_consensus.cc:348] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:46.910953 4893 raft_consensus.cc:374] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:46.911118 4893 raft_consensus.cc:729] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1d0d70d38444e7b8729d86e1445763, State: Initialized, Role: FOLLOWER
I20260807 02:55:46.911528 4892 ts_tablet_manager.cc:1428] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 02:55:46.911475 4893 consensus_queue.cc:260] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:46.913276 4893 ts_tablet_manager.cc:1428] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260807 02:55:47.314776 4901 raft_consensus.cc:482] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:47.315025 4901 raft_consensus.cc:504] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.315781 4901 leader_election.cc:288] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347), 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:47.316431 4300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84392126894c45d1a91032e79d417ec6" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1d0d70d38444e7b8729d86e1445763" is_pre_election: true
I20260807 02:55:47.316430 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84392126894c45d1a91032e79d417ec6" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82" is_pre_election: true
I20260807 02:55:47.316720 4678 raft_consensus.cc:2454] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in term 0.
I20260807 02:55:47.316789 4300 raft_consensus.cc:2454] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in term 0.
I20260807 02:55:47.317339 4388 leader_election.cc:302] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 78f127347f7049f5b7f1be70b2811f82, 9896e83f5a264d30b3dbcf0cf3c4ad44, 9d1d0d70d38444e7b8729d86e1445763; no voters:
I20260807 02:55:47.317693 4901 raft_consensus.cc:2781] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:55:47.317837 4901 raft_consensus.cc:482] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:55:47.317957 4901 raft_consensus.cc:3037] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:47.320160 4901 raft_consensus.cc:504] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.320758 4901 leader_election.cc:288] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 election: Requested vote from peers 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347), 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:47.321412 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84392126894c45d1a91032e79d417ec6" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82"
I20260807 02:55:47.321460 4300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84392126894c45d1a91032e79d417ec6" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1d0d70d38444e7b8729d86e1445763"
I20260807 02:55:47.321615 4678 raft_consensus.cc:3037] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:47.321753 4300 raft_consensus.cc:3037] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:47.323768 4300 raft_consensus.cc:2454] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in term 1.
I20260807 02:55:47.324150 4678 raft_consensus.cc:2454] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in term 1.
I20260807 02:55:47.324236 4388 leader_election.cc:302] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9896e83f5a264d30b3dbcf0cf3c4ad44, 9d1d0d70d38444e7b8729d86e1445763; no voters:
I20260807 02:55:47.324565 4901 raft_consensus.cc:2781] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:47.325073 4901 raft_consensus.cc:686] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 LEADER]: Becoming Leader. State: Replica: 9896e83f5a264d30b3dbcf0cf3c4ad44, State: Running, Role: LEADER
I20260807 02:55:47.325536 4901 consensus_queue.cc:237] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.328682 4084 catalog_manager.cc:5495] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66). New cstate: current_term: 1 leader_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260807 02:55:47.372689 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:47.365967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.1.121.66:44555)
0807 02:55:47.366556 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:47.366560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:47.366580 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:55:47.366678 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:47.366681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:47.366753 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:47.366895 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:47.367419 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.368244 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.370061 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.370665 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.370767 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:47.371296 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:47.371937 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:47.372047 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:47.372232 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 02:55:47.372338 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":327,"threads_started":1}
I20260807 02:55:47.389328 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:47.383803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:52439 (local address 127.1.121.65:38347)
0807 02:55:47.384545 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:47.384549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:47.384571 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:55:47.384629 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:47.384633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:47.384719 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:47.384803 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:47.385465 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.386235 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.387713 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.388081 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.388168 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:47.388660 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:47.388754 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:47.388820 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:47.389011 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 02:55:47.389071 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":492,"threads_started":1}
I20260807 02:55:47.389395 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:47.383558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38347 (local address 127.1.121.66:52439)
0807 02:55:47.384291 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:47.384308 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:47.384415 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:47.384880 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:47.384886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:47.384905 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:55:47.385304 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:47.385314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.386373 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.386377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:47.387591 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:47.387604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.388227 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.388232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:47.388394 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:47.388460 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:55:47.388897 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:55:47.388906 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 02:55:47.388971 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 02:55:47.389074 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":331,"threads_started":1}
I20260807 02:55:47.398867 4678 raft_consensus.cc:1261] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Refusing update from remote peer 9896e83f5a264d30b3dbcf0cf3c4ad44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:47.399191 4300 raft_consensus.cc:1261] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Refusing update from remote peer 9896e83f5a264d30b3dbcf0cf3c4ad44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:47.399638 4902 consensus_queue.cc:1035] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:47.400219 4901 consensus_queue.cc:1035] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:47.404193 4908 mvcc.cc:205] Tried to move back new op lower bound from 7315748238939471872 to 7315748238648520704. Current Snapshot: MvccSnapshot[applied={T|T < 7315748238939471872}]
I20260807 02:55:47.432173 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:47.424889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:54825 (local address 127.1.121.66:44555)
0807 02:55:47.425047 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:47.425053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:47.427021 (+ 1968us) server_negotiation.cc:407] Connection header received
0807 02:55:47.427296 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:47.427302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:47.427383 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:47.427487 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:55:47.427888 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.428729 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.430199 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.430662 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.430834 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:47.431368 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:55:47.431496 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:55:47.431604 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:47.431752 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 02:55:47.431831 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 02:55:47.432374 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:47.424911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:44555 (local address 127.1.121.65:54825)
0807 02:55:47.426912 (+ 2001us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:47.426928 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:47.427092 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:47.427534 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:47.427538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:47.427547 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:55:47.427738 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:47.427745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.428903 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.428911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:47.430052 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:47.430066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:47.430828 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:47.430835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:47.431122 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:47.431175 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:55:47.431613 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:55:47.431628 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 02:55:47.431719 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 02:55:47.431899 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1811,"thread_start_us":1653,"threads_started":1}
I20260807 02:55:47.467609 4085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:47.467887 4085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:47.471402 4085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71: Sending DeleteTablet for 3 replicas of tablet 84392126894c45d1a91032e79d417ec6
I20260807 02:55:47.472287 4264 tablet_service.cc:1514] Processing DeleteTablet for tablet 84392126894c45d1a91032e79d417ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:47 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:51140
I20260807 02:55:47.472623 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 84392126894c45d1a91032e79d417ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:47 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:43790
I20260807 02:55:47.473183 4455 tablet_service.cc:1514] Processing DeleteTablet for tablet 84392126894c45d1a91032e79d417ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:47 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:48660
I20260807 02:55:47.473389 4930 tablet_replica.cc:330] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: stopping tablet replica
I20260807 02:55:47.473637 4930 raft_consensus.cc:2229] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:47.474071 4930 raft_consensus.cc:2258] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:47.474400 4932 tablet_replica.cc:330] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: stopping tablet replica
I20260807 02:55:47.474452 4085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 02:55:47.474670 4932 raft_consensus.cc:2229] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:47.475020 4932 raft_consensus.cc:2258] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 02:55:47.475163 4085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:55:47.475772 4930 ts_tablet_manager.cc:1905] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:47.476066 4932 ts_tablet_manager.cc:1905] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:47.483455 4932 ts_tablet_manager.cc:1918] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:55:47.483633 4932 log.cc:1196] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/84392126894c45d1a91032e79d417ec6
I20260807 02:55:47.484225 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 89c315e683f84770a1cb33865b8e7048 (DEFAULT_TABLE table=test-table [id=5e7c12525de04bf8b90197fd7ff2879d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:47.484295 4264 tablet_service.cc:1467] Processing CreateTablet for tablet 89c315e683f84770a1cb33865b8e7048 (DEFAULT_TABLE table=test-table [id=5e7c12525de04bf8b90197fd7ff2879d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:47.484686 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c315e683f84770a1cb33865b8e7048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:47.484802 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c315e683f84770a1cb33865b8e7048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:47.485119 4453 tablet_service.cc:1467] Processing CreateTablet for tablet 89c315e683f84770a1cb33865b8e7048 (DEFAULT_TABLE table=test-table [id=5e7c12525de04bf8b90197fd7ff2879d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:55:47.485872 4453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c315e683f84770a1cb33865b8e7048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:47.491850 4932 ts_tablet_manager.cc:1939] T 84392126894c45d1a91032e79d417ec6 P 78f127347f7049f5b7f1be70b2811f82: Deleting consensus metadata
I20260807 02:55:47.493448 4066 catalog_manager.cc:4841] TS 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003): tablet 84392126894c45d1a91032e79d417ec6 (table test-table [id=e21b35b94ec34f34b96d118809ba70c5]) successfully deleted
I20260807 02:55:47.499421 4940 tablet_bootstrap.cc:492] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Bootstrap starting.
I20260807 02:55:47.500247 4938 tablet_bootstrap.cc:492] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: Bootstrap starting.
I20260807 02:55:47.500715 4936 tablet_replica.cc:330] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: stopping tablet replica
I20260807 02:55:47.501621 4937 tablet_bootstrap.cc:492] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: Bootstrap starting.
I20260807 02:55:47.502044 4936 raft_consensus.cc:2229] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:55:47.503058 4938 tablet_bootstrap.cc:654] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:47.504733 4937 tablet_bootstrap.cc:654] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:47.504916 4936 raft_consensus.cc:2258] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:47.505903 4930 ts_tablet_manager.cc:1918] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:55:47.506062 4930 log.cc:1196] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/84392126894c45d1a91032e79d417ec6
I20260807 02:55:47.506701 4930 ts_tablet_manager.cc:1939] T 84392126894c45d1a91032e79d417ec6 P 9d1d0d70d38444e7b8729d86e1445763: Deleting consensus metadata
I20260807 02:55:47.507244 4936 ts_tablet_manager.cc:1905] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:47.507829 4064 catalog_manager.cc:4841] TS 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347): tablet 84392126894c45d1a91032e79d417ec6 (table test-table [id=e21b35b94ec34f34b96d118809ba70c5]) successfully deleted
I20260807 02:55:47.508805 4938 tablet_bootstrap.cc:492] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: No bootstrap required, opened a new log
I20260807 02:55:47.509011 4940 tablet_bootstrap.cc:654] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:47.509035 4938 ts_tablet_manager.cc:1397] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260807 02:55:47.510097 4938 raft_consensus.cc:348] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.510363 4938 raft_consensus.cc:374] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:47.510481 4938 raft_consensus.cc:729] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f127347f7049f5b7f1be70b2811f82, State: Initialized, Role: FOLLOWER
I20260807 02:55:47.510833 4938 consensus_queue.cc:260] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.511631 4938 ts_tablet_manager.cc:1428] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 02:55:47.518600 4937 tablet_bootstrap.cc:492] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: No bootstrap required, opened a new log
I20260807 02:55:47.518857 4937 ts_tablet_manager.cc:1397] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.011s
I20260807 02:55:47.519994 4937 raft_consensus.cc:348] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.520437 4937 raft_consensus.cc:374] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:47.520601 4937 raft_consensus.cc:729] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1d0d70d38444e7b8729d86e1445763, State: Initialized, Role: FOLLOWER
I20260807 02:55:47.521021 4937 consensus_queue.cc:260] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.521944 4937 ts_tablet_manager.cc:1428] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:55:47.527479 4936 ts_tablet_manager.cc:1918] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:55:47.527767 4936 log.cc:1196] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/84392126894c45d1a91032e79d417ec6
I20260807 02:55:47.528645 4936 ts_tablet_manager.cc:1939] T 84392126894c45d1a91032e79d417ec6 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting consensus metadata
I20260807 02:55:47.530112 4067 catalog_manager.cc:4841] TS 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555): tablet 84392126894c45d1a91032e79d417ec6 (table test-table [id=e21b35b94ec34f34b96d118809ba70c5]) successfully deleted
I20260807 02:55:47.530421 4940 tablet_bootstrap.cc:492] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: No bootstrap required, opened a new log
I20260807 02:55:47.530604 4940 ts_tablet_manager.cc:1397] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.012s
I20260807 02:55:47.531600 4940 raft_consensus.cc:348] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.531963 4940 raft_consensus.cc:374] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:47.532184 4940 raft_consensus.cc:729] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9896e83f5a264d30b3dbcf0cf3c4ad44, State: Initialized, Role: FOLLOWER
I20260807 02:55:47.532466 4940 consensus_queue.cc:260] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.533232 4940 ts_tablet_manager.cc:1428] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 02:55:47.627106 4901 raft_consensus.cc:482] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:55:47.627382 4901 raft_consensus.cc:504] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.628305 4901 leader_election.cc:288] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347), 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:47.629086 4300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c315e683f84770a1cb33865b8e7048" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1d0d70d38444e7b8729d86e1445763" is_pre_election: true
I20260807 02:55:47.629129 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c315e683f84770a1cb33865b8e7048" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82" is_pre_election: true
I20260807 02:55:47.629433 4678 raft_consensus.cc:2454] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in term 0.
I20260807 02:55:47.629415 4300 raft_consensus.cc:2454] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in term 0.
I20260807 02:55:47.629961 4389 leader_election.cc:302] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f127347f7049f5b7f1be70b2811f82, 9896e83f5a264d30b3dbcf0cf3c4ad44; no voters:
I20260807 02:55:47.630367 4901 raft_consensus.cc:2781] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:55:47.630529 4901 raft_consensus.cc:482] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:55:47.630657 4901 raft_consensus.cc:3037] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:47.634347 4901 raft_consensus.cc:504] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.635102 4901 leader_election.cc:288] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 election: Requested vote from peers 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347), 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003)
I20260807 02:55:47.635910 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c315e683f84770a1cb33865b8e7048" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f127347f7049f5b7f1be70b2811f82"
I20260807 02:55:47.636175 4678 raft_consensus.cc:3037] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:47.636037 4300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c315e683f84770a1cb33865b8e7048" candidate_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1d0d70d38444e7b8729d86e1445763"
I20260807 02:55:47.636449 4300 raft_consensus.cc:3037] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:47.639680 4300 raft_consensus.cc:2454] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in term 1.
I20260807 02:55:47.639739 4678 raft_consensus.cc:2454] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9896e83f5a264d30b3dbcf0cf3c4ad44 in term 1.
I20260807 02:55:47.640282 4389 leader_election.cc:302] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 78f127347f7049f5b7f1be70b2811f82, 9896e83f5a264d30b3dbcf0cf3c4ad44, 9d1d0d70d38444e7b8729d86e1445763; no voters:
I20260807 02:55:47.640697 4901 raft_consensus.cc:2781] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:47.640887 4901 raft_consensus.cc:686] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 LEADER]: Becoming Leader. State: Replica: 9896e83f5a264d30b3dbcf0cf3c4ad44, State: Running, Role: LEADER
I20260807 02:55:47.641206 4901 consensus_queue.cc:237] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } }
I20260807 02:55:47.644806 4082 catalog_manager.cc:5495] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66). New cstate: current_term: 1 leader_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9896e83f5a264d30b3dbcf0cf3c4ad44" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260807 02:55:47.687602 4678 raft_consensus.cc:1261] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Refusing update from remote peer 9896e83f5a264d30b3dbcf0cf3c4ad44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:47.688042 4300 raft_consensus.cc:1261] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Refusing update from remote peer 9896e83f5a264d30b3dbcf0cf3c4ad44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:55:47.688306 4901 consensus_queue.cc:1035] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f127347f7049f5b7f1be70b2811f82" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:55:47.688645 4902 consensus_queue.cc:1035] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1d0d70d38444e7b8729d86e1445763" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 02:55:47.726460 4908 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 02:55:49.326637 4082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:49.326900 4082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:55:49.330981 4082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2de93b5fb8a2465192593c43f2541e71: Sending DeleteTablet for 3 replicas of tablet 89c315e683f84770a1cb33865b8e7048
I20260807 02:55:49.331837 4264 tablet_service.cc:1514] Processing DeleteTablet for tablet 89c315e683f84770a1cb33865b8e7048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:49 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:51140
I20260807 02:55:49.331915 4455 tablet_service.cc:1514] Processing DeleteTablet for tablet 89c315e683f84770a1cb33865b8e7048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:49 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:48660
I20260807 02:55:49.332157 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 89c315e683f84770a1cb33865b8e7048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:55:49 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:43790
I20260807 02:55:49.333061 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 4177
I20260807 02:55:49.336916 4981 tablet_replica.cc:330] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: stopping tablet replica
I20260807 02:55:49.337256 4981 raft_consensus.cc:2229] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:49.337826 4981 raft_consensus.cc:2258] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:49.338541 4979 tablet_replica.cc:330] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: stopping tablet replica
I20260807 02:55:49.338907 4979 raft_consensus.cc:2229] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:55:49.339293 4979 raft_consensus.cc:2258] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:49.339309 4981 ts_tablet_manager.cc:1905] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:49.340382 4979 ts_tablet_manager.cc:1905] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:49.340963 4978 tablet_replica.cc:330] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: stopping tablet replica
I20260807 02:55:49.341166 4978 raft_consensus.cc:2229] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:55:49.341398 4978 raft_consensus.cc:2258] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:55:49.342330 4978 ts_tablet_manager.cc:1905] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:55:49.346975 4981 ts_tablet_manager.cc:1918] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:55:49.347174 4981 log.cc:1196] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/89c315e683f84770a1cb33865b8e7048
I20260807 02:55:49.347692 4981 ts_tablet_manager.cc:1939] T 89c315e683f84770a1cb33865b8e7048 P 78f127347f7049f5b7f1be70b2811f82: Deleting consensus metadata
I20260807 02:55:49.348155 4979 ts_tablet_manager.cc:1918] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:55:49.348367 4979 log.cc:1196] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/89c315e683f84770a1cb33865b8e7048
I20260807 02:55:49.349025 4066 catalog_manager.cc:4841] TS 78f127347f7049f5b7f1be70b2811f82 (127.1.121.67:39003): tablet 89c315e683f84770a1cb33865b8e7048 (table test-table [id=5e7c12525de04bf8b90197fd7ff2879d]) successfully deleted
I20260807 02:55:49.349133 4979 ts_tablet_manager.cc:1939] T 89c315e683f84770a1cb33865b8e7048 P 9896e83f5a264d30b3dbcf0cf3c4ad44: Deleting consensus metadata
I20260807 02:55:49.350432 4067 catalog_manager.cc:4841] TS 9896e83f5a264d30b3dbcf0cf3c4ad44 (127.1.121.66:44555): tablet 89c315e683f84770a1cb33865b8e7048 (table test-table [id=5e7c12525de04bf8b90197fd7ff2879d]) successfully deleted
I20260807 02:55:49.350518 4978 ts_tablet_manager.cc:1918] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:55:49.350711 4978 log.cc:1196] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/89c315e683f84770a1cb33865b8e7048
Aug 07 02:55:49 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071346, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:55:49.351260 4978 ts_tablet_manager.cc:1939] T 89c315e683f84770a1cb33865b8e7048 P 9d1d0d70d38444e7b8729d86e1445763: Deleting consensus metadata
I20260807 02:55:49.352293 4064 catalog_manager.cc:4841] TS 9d1d0d70d38444e7b8729d86e1445763 (127.1.121.65:38347): tablet 89c315e683f84770a1cb33865b8e7048 (table test-table [id=5e7c12525de04bf8b90197fd7ff2879d]) successfully deleted
I20260807 02:55:49.358343 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:49.334190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.1.121.65:38347)
0807 02:55:49.340030 (+ 5840us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:49.340038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:49.340075 (+ 37us) server_negotiation.cc:407] Connection header received
0807 02:55:49.340135 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:49.340141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:49.340225 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:49.340362 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:49.342201 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:49.343391 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:49.345517 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:49.346155 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:49.346290 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:49.352020 (+ 5730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:49.352036 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:49.352054 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:49.352089 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:49.355293 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:49.356032 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:49.356040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:49.356043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:49.356194 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:49.356619 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:49.356630 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:49.356632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:49.357075 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:49.357447 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:49.357690 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 02:55:49.357926 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5695,"thread_start_us":5563,"threads_started":1}
W20260807 02:55:49.359028 4353 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46382
W20260807 02:55:49.359717 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 4177: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 02:55:49.359949 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 4177
I20260807 02:55:49.397894 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 4367
Aug 07 02:55:49 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071346, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:55:49.414328 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:49.398753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.1.121.66:44555)
0807 02:55:49.399492 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:49.399496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:49.399513 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:55:49.399572 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:49.399575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:49.399633 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:49.399758 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:49.401498 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:49.402277 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:49.403549 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:49.404108 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:49.404208 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:49.408557 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:49.408573 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:49.408591 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:49.408628 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:49.411477 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:49.412263 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:49.412270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:49.412272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:49.412457 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:49.412901 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:49.412905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:49.412907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:49.413359 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:49.413617 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:49.413876 (+ 259us) server_negotiation.cc:299] Negotiation successful
0807 02:55:49.414051 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":356,"threads_started":1}
W20260807 02:55:49.415208 4543 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45532
W20260807 02:55:49.415902 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 4367: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 02:55:49.416074 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 4367
I20260807 02:55:49.462476 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 4556
Aug 07 02:55:49 dist-test-slave-3h5h krb5kdc[4017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071346, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:55:49.478916 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:49.463344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.1.121.67:39003)
0807 02:55:49.463977 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:49.463982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:49.463999 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:55:49.464058 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:49.464061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:49.464153 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:49.464254 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:49.465683 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:49.466464 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:49.467648 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:49.468076 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:49.468170 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:49.472627 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:49.472645 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:49.472663 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:49.472698 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:49.475903 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:49.476593 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:49.476601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:49.476604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:49.476777 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:49.477205 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:49.477209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:49.477212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:49.477697 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:49.477940 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:49.478225 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 02:55:49.478447 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":381,"threads_started":1}
W20260807 02:55:49.479768 4736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55278
W20260807 02:55:49.480489 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 4556: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 02:55:49.480703 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 4556
I20260807 02:55:49.505520 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 4042
I20260807 02:55:49.507012 4136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:49.629866 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 4042
2026-08-07T02:55:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10562 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:55:49 dist-test-slave-3h5h krb5kdc[4993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:55:49 dist-test-slave-3h5h krb5kdc[4993](info): set up 2 sockets
Aug 07 02:55:49 dist-test-slave-3h5h krb5kdc[4993](info): commencing operation
krb5kdc: starting...
W20260807 02:55:50.911252 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:55:51 dist-test-slave-3h5h krb5kdc[4993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:55:51.149837 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37329
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:51.490141 5010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:51.490424 5010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:51.490590 5010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:51.497845 5010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:55:51.497957 5010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:51.498054 5010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:51.498167 5010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:55:51.498251 5010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:55:51.509024 5010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:37329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:51.510439 5010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:51.511828 5010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:51.524595 5018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:51.524608 5021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:51.524585 5019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:51.525527 5010 server_base.cc:1034] running on GCE node
I20260807 02:55:51.526364 5010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:51.527720 5010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:51.529002 5010 hybrid_clock.cc:648] HybridClock initialized: now 1786071351528879 us; error 88 us; skew 500 ppm
I20260807 02:55:51.529481 5010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:51 dist-test-slave-3h5h krb5kdc[4993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071351, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:51.533108 5010 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:55:51.534718 5010 webserver.cc:460] Webserver started at http://127.1.121.126:36527/ using document root <none> and password file <none>
I20260807 02:55:51.535501 5010 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:51.535676 5010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:51.536049 5010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:51.539564 5010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "8afcfe0189264b49aca734c3e0286593"
format_stamp: "Formatted at 2026-08-07 02:55:51 on dist-test-slave-3h5h"
server_key: "98cfcd0da1f3e21f005d5beb76bd2a8f"
server_key_iv: "bb28b72d70fbe19bcd197f8a1ef9b155"
server_key_version: "encryptionkey@0"
I20260807 02:55:51.540489 5010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "8afcfe0189264b49aca734c3e0286593"
format_stamp: "Formatted at 2026-08-07 02:55:51 on dist-test-slave-3h5h"
server_key: "98cfcd0da1f3e21f005d5beb76bd2a8f"
server_key_iv: "bb28b72d70fbe19bcd197f8a1ef9b155"
server_key_version: "encryptionkey@0"
I20260807 02:55:51.546555 5010 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260807 02:55:51.550511 5028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:51.552027 5010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:55:51.552253 5010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "8afcfe0189264b49aca734c3e0286593"
format_stamp: "Formatted at 2026-08-07 02:55:51 on dist-test-slave-3h5h"
server_key: "98cfcd0da1f3e21f005d5beb76bd2a8f"
server_key_iv: "bb28b72d70fbe19bcd197f8a1ef9b155"
server_key_version: "encryptionkey@0"
I20260807 02:55:51.552533 5010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:51.574257 5010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:51.578797 5010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:51.579121 5010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:51.599944 5010 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:37329
I20260807 02:55:51.599963 5105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:37329 every 8 connection(s)
I20260807 02:55:51.601581 5010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:55:51.603358 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 5010
I20260807 02:55:51.603675 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:55:51.604743 1509 external_mini_cluster.cc:1428] Setting key b2e5e7278bd9c8352a7771c15c9700a5
I20260807 02:55:51.607581 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 02:55:51 dist-test-slave-3h5h krb5kdc[4993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:51.623523 5106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593: Bootstrap starting.
I20260807 02:55:51.627485 5106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:51.628944 5106 log.cc:824] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:51.631649 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:51.607296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.1.121.126:37329)
0807 02:55:51.608184 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:51.608197 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:51.608240 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:55:51.610031 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:51.610055 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:51.610435 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:51.610774 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:51.613185 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:51.614731 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:51.616918 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:51.617721 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:51.617863 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:51.623433 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:51.623442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:51.623458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:51.623508 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:51.626991 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:51.627816 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:51.627828 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:51.627835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:51.628082 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:51.628557 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:51.628563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:51.628567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:51.629270 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:51.629615 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:51.630142 (+ 527us) server_negotiation.cc:299] Negotiation successful
0807 02:55:51.630615 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":439,"threads_started":1}
I20260807 02:55:51.632525 5106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593: No bootstrap required, opened a new log
I20260807 02:55:51.641983 5106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcfe0189264b49aca734c3e0286593" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37329 } }
I20260807 02:55:51.642458 5106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:51.642601 5106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afcfe0189264b49aca734c3e0286593, State: Initialized, Role: FOLLOWER
I20260807 02:55:51.643431 5106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcfe0189264b49aca734c3e0286593" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37329 } }
I20260807 02:55:51.643777 5106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:55:51.643961 5106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:55:51.644192 5106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:51.646869 5106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcfe0189264b49aca734c3e0286593" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37329 } }
I20260807 02:55:51.647487 5106 leader_election.cc:302] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8afcfe0189264b49aca734c3e0286593; no voters:
I20260807 02:55:51.648267 5106 leader_election.cc:288] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:55:51.648509 5112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:51.649718 5112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [term 1 LEADER]: Becoming Leader. State: Replica: 8afcfe0189264b49aca734c3e0286593, State: Running, Role: LEADER
I20260807 02:55:51.650359 5112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcfe0189264b49aca734c3e0286593" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37329 } }
I20260807 02:55:51.651366 5106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:55:51.653604 5113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8afcfe0189264b49aca734c3e0286593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcfe0189264b49aca734c3e0286593" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37329 } } }
I20260807 02:55:51.654000 5113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:51.653957 5114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8afcfe0189264b49aca734c3e0286593. Latest consensus state: current_term: 1 leader_uuid: "8afcfe0189264b49aca734c3e0286593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcfe0189264b49aca734c3e0286593" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37329 } } }
I20260807 02:55:51.654407 5114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:51.659215 5123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:55:51.665735 5123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:55:51.675285 5123 catalog_manager.cc:1347] Generated new cluster ID: a03207b38a5c4f08baf566360d423856
I20260807 02:55:51.675414 5123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:55:51.717753 5123 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:55:51.719239 5123 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:55:51.726919 5123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8afcfe0189264b49aca734c3e0286593: Generated new TSK 0
I20260807 02:55:51.727794 5123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:55:51.804983 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37329
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:52.159274 5148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:52.159650 5148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:52.159763 5148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:52.167754 5148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:52.167878 5148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:52.168004 5148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:55:52.178470 5148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:52.179807 5148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:52.181308 5148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:52.196282 5154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:52.196337 5155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:52.196237 5157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:52.197706 5148 server_base.cc:1034] running on GCE node
I20260807 02:55:52.198490 5148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:52.199573 5148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:52.200906 5148 hybrid_clock.cc:648] HybridClock initialized: now 1786071352200804 us; error 50 us; skew 500 ppm
I20260807 02:55:52.201313 5148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:52 dist-test-slave-3h5h krb5kdc[4993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071352, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:52.207917 5148 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:55:52.209570 5148 webserver.cc:460] Webserver started at http://127.1.121.65:37005/ using document root <none> and password file <none>
I20260807 02:55:52.210350 5148 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:52.210500 5148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:52.210909 5148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:52.214118 5148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "265c6b0ff40f4bf68ad8f3c7f865bd3f"
format_stamp: "Formatted at 2026-08-07 02:55:52 on dist-test-slave-3h5h"
server_key: "15202ea925ce08271953f46dd0535bc0"
server_key_iv: "866729d8dd284472947a87785e87931a"
server_key_version: "encryptionkey@0"
I20260807 02:55:52.214984 5148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "265c6b0ff40f4bf68ad8f3c7f865bd3f"
format_stamp: "Formatted at 2026-08-07 02:55:52 on dist-test-slave-3h5h"
server_key: "15202ea925ce08271953f46dd0535bc0"
server_key_iv: "866729d8dd284472947a87785e87931a"
server_key_version: "encryptionkey@0"
I20260807 02:55:52.220476 5148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 02:55:52.224025 5168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:52.225544 5148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 02:55:52.225733 5148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "265c6b0ff40f4bf68ad8f3c7f865bd3f"
format_stamp: "Formatted at 2026-08-07 02:55:52 on dist-test-slave-3h5h"
server_key: "15202ea925ce08271953f46dd0535bc0"
server_key_iv: "866729d8dd284472947a87785e87931a"
server_key_version: "encryptionkey@0"
I20260807 02:55:52.226011 5148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:52.247552 5148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:52.252792 5148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:52.253139 5148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:52.254257 5148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:52.255767 5148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:52.255863 5148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:52.255956 5148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:52.256013 5148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:52.294427 5148 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:38525
I20260807 02:55:52.294528 5328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:38525 every 8 connection(s)
I20260807 02:55:52.295944 5148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:55:52.298635 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 5148
I20260807 02:55:52.298879 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:55:52.299737 1509 external_mini_cluster.cc:1428] Setting key 3f0a04830fe4220d3379de47fa7971ea
Aug 07 02:55:52 dist-test-slave-3h5h krb5kdc[4993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071352, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:52.320815 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:52.298921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:40151 (local address 127.1.121.126:37329)
0807 02:55:52.299449 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:52.299457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:52.300871 (+ 1414us) server_negotiation.cc:407] Connection header received
0807 02:55:52.302953 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:52.302962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:52.303059 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:52.303208 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:52.304890 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:52.306143 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:52.307882 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:52.308715 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:52.308916 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:52.313185 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:52.313195 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:52.313203 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:52.313252 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:52.317160 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:52.317962 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:52.317970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:52.317976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:52.318150 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:52.318664 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:52.318672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:52.318678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:52.319083 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:52.319245 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:52.320098 (+ 853us) server_negotiation.cc:299] Negotiation successful
0807 02:55:52.320368 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":240,"threads_started":1}
I20260807 02:55:52.321904 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:52.299599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37329 (local address 127.1.121.65:40151)
0807 02:55:52.300638 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:52.300707 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:52.302649 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:52.303430 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:52.303438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:52.304025 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:52.304657 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:52.304677 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:52.306331 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:52.306335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:52.307710 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:52.307724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:52.309002 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:52.309007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:52.309206 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:52.309945 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:52.309973 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:52.312928 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:52.317314 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:52.317322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:52.317337 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:52.317829 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:52.318304 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:52.318308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:52.318311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:52.318501 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:52.319223 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:52.319230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:52.319592 (+ 362us) client_negotiation.cc:769] Sending connection context
0807 02:55:52.319948 (+ 356us) client_negotiation.cc:240] Negotiation successful
0807 02:55:52.320519 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":497,"threads_started":1}
I20260807 02:55:52.323632 5329 heartbeater.cc:347] Connected to a master server at 127.1.121.126:37329
I20260807 02:55:52.324011 5329 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
I20260807 02:55:52.324971 5329 heartbeater.cc:511] Master 127.1.121.126:37329 requested a full tablet report, sending...
I20260807 02:55:52.327229 5054 ts_manager.cc:194] Registered new tserver with Master: 265c6b0ff40f4bf68ad8f3c7f865bd3f (127.1.121.65:38525)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
I20260807 02:55:52.329011 5054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:40151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:55:52.356297 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37329
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:52.715441 5341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:52.715795 5341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:52.715999 5341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:52.723693 5341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:52.723861 5341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:52.724056 5341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:55:52.734867 5341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:52.736348 5341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:52.738380 5341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:52.753198 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:52.753448 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:52.753497 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:52.754818 5341 server_base.cc:1034] running on GCE node
I20260807 02:55:52.755473 5341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:52.756449 5341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:52.757690 5341 hybrid_clock.cc:648] HybridClock initialized: now 1786071352757645 us; error 35 us; skew 500 ppm
I20260807 02:55:52.758051 5341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:52 dist-test-slave-3h5h krb5kdc[4993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071352, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:52.761734 5341 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:55:52.763428 5341 webserver.cc:460] Webserver started at http://127.1.121.66:39181/ using document root <none> and password file <none>
I20260807 02:55:52.764197 5341 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:52.764360 5341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:52.764708 5341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:52.768056 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "fdb010b871224373b7e424dfe116e4a2"
format_stamp: "Formatted at 2026-08-07 02:55:52 on dist-test-slave-3h5h"
server_key: "b0bd54ef8f1e17a5e88e02dda8fe8c29"
server_key_iv: "f93ec52e3f679333c492e471e022ff6c"
server_key_version: "encryptionkey@0"
I20260807 02:55:52.769048 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "fdb010b871224373b7e424dfe116e4a2"
format_stamp: "Formatted at 2026-08-07 02:55:52 on dist-test-slave-3h5h"
server_key: "b0bd54ef8f1e17a5e88e02dda8fe8c29"
server_key_iv: "f93ec52e3f679333c492e471e022ff6c"
server_key_version: "encryptionkey@0"
I20260807 02:55:52.774770 5341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:55:52.778451 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:52.779814 5341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:55:52.780001 5341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "fdb010b871224373b7e424dfe116e4a2"
format_stamp: "Formatted at 2026-08-07 02:55:52 on dist-test-slave-3h5h"
server_key: "b0bd54ef8f1e17a5e88e02dda8fe8c29"
server_key_iv: "f93ec52e3f679333c492e471e022ff6c"
server_key_version: "encryptionkey@0"
I20260807 02:55:52.780215 5341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:52.799374 5341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:52.803612 5341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:52.803978 5341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:52.805054 5341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:52.806515 5341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:52.806622 5341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:52.806784 5341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:52.806860 5341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:52.842464 5341 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:38015
I20260807 02:55:52.842483 5522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:38015 every 8 connection(s)
I20260807 02:55:52.843780 5341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:55:52.850122 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 5341
I20260807 02:55:52.850337 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:55:52.850927 1509 external_mini_cluster.cc:1428] Setting key 9a977ec5a5343d8fc2a428f782d4a603
Aug 07 02:55:52 dist-test-slave-3h5h krb5kdc[4993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071352, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:52.864789 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:52.846400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:46139 (local address 127.1.121.126:37329)
0807 02:55:52.846878 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:52.846884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:52.847936 (+ 1052us) server_negotiation.cc:407] Connection header received
0807 02:55:52.849951 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:52.849955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:52.850025 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:52.850122 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:52.852249 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:52.853274 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:52.854814 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:52.855259 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:52.855391 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:52.859072 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:52.859077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:52.859081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:52.859113 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:52.861760 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:52.862490 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:52.862496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:52.862504 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:52.862625 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:52.863046 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:52.863053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:52.863056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:52.863331 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:52.863485 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:52.864175 (+ 690us) server_negotiation.cc:299] Negotiation successful
0807 02:55:52.864394 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":237,"threads_started":1}
I20260807 02:55:52.865796 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:52.847011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37329 (local address 127.1.121.66:46139)
0807 02:55:52.847761 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:52.847799 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:52.849628 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:52.850439 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:52.850450 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:52.851216 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:52.852029 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:52.852057 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:52.853435 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:52.853439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:52.854663 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:52.854676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:52.855505 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:52.855512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:52.855742 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:52.856434 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:52.856460 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:52.858826 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:52.861902 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:52.861910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:52.861928 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:52.862357 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:52.862739 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:52.862745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:52.862748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:52.862932 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:52.863459 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:52.863467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:52.863806 (+ 339us) client_negotiation.cc:769] Sending connection context
0807 02:55:52.864094 (+ 288us) client_negotiation.cc:240] Negotiation successful
0807 02:55:52.864611 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":295,"threads_started":1}
I20260807 02:55:52.867266 5525 heartbeater.cc:347] Connected to a master server at 127.1.121.126:37329
I20260807 02:55:52.867607 5525 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:52.868588 5525 heartbeater.cc:511] Master 127.1.121.126:37329 requested a full tablet report, sending...
I20260807 02:55:52.870373 5054 ts_manager.cc:194] Registered new tserver with Master: fdb010b871224373b7e424dfe116e4a2 (127.1.121.66:38015)
I20260807 02:55:52.871251 5054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:46139
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:55:52.906243 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37329
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:53.259829 5535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:53.260273 5535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:53.260474 5535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:53.268528 5535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:53.268692 5535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:53.268904 5535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:55:53.280157 5535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:53.281674 5535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:53.283139 5535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:53.299229 5541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:53.302258 5543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:53.303032 5546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:53.304474 5535 server_base.cc:1034] running on GCE node
I20260807 02:55:53.305465 5535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:53.306785 5535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:53.309088 5535 hybrid_clock.cc:648] HybridClock initialized: now 1786071353308996 us; error 66 us; skew 500 ppm
I20260807 02:55:53.309597 5535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:53 dist-test-slave-3h5h krb5kdc[4993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071353, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:53.314289 5535 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:55:53.316448 5535 webserver.cc:460] Webserver started at http://127.1.121.67:39735/ using document root <none> and password file <none>
I20260807 02:55:53.317528 5535 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:53.317683 5535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:53.318079 5535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:53.322561 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "ec8aedd23b05490397ecce81d70a658a"
format_stamp: "Formatted at 2026-08-07 02:55:53 on dist-test-slave-3h5h"
server_key: "1d03947eff5d77fd400b52ebd59f0ca0"
server_key_iv: "a217b77e6a3dc5f54dcdc3d867960086"
server_key_version: "encryptionkey@0"
I20260807 02:55:53.323770 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "ec8aedd23b05490397ecce81d70a658a"
format_stamp: "Formatted at 2026-08-07 02:55:53 on dist-test-slave-3h5h"
server_key: "1d03947eff5d77fd400b52ebd59f0ca0"
server_key_iv: "a217b77e6a3dc5f54dcdc3d867960086"
server_key_version: "encryptionkey@0"
I20260807 02:55:53.332190 5329 heartbeater.cc:503] Master 127.1.121.126:37329 was elected leader, sending a full tablet report...
I20260807 02:55:53.332311 5535 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260807 02:55:53.336575 5555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:53.338019 5535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:55:53.338191 5535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "ec8aedd23b05490397ecce81d70a658a"
format_stamp: "Formatted at 2026-08-07 02:55:53 on dist-test-slave-3h5h"
server_key: "1d03947eff5d77fd400b52ebd59f0ca0"
server_key_iv: "a217b77e6a3dc5f54dcdc3d867960086"
server_key_version: "encryptionkey@0"
I20260807 02:55:53.338413 5535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:53.353936 5535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:53.358021 5535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:53.358350 5535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:53.359429 5535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:53.360926 5535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:53.361004 5535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:53.361097 5535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:53.361145 5535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:53.395979 5535 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:43819
I20260807 02:55:53.396023 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:43819 every 8 connection(s)
I20260807 02:55:53.397408 5535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:55:53.402314 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 5535
I20260807 02:55:53.402520 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:55:53.403060 1509 external_mini_cluster.cc:1428] Setting key 3729be54d5775dd76a2178c1ffb5268a
Aug 07 02:55:53 dist-test-slave-3h5h krb5kdc[4993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071353, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:53.418936 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:53.400144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:46577 (local address 127.1.121.126:37329)
0807 02:55:53.400631 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:53.400635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:53.401691 (+ 1056us) server_negotiation.cc:407] Connection header received
0807 02:55:53.404193 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:53.404198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:53.404265 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:53.404354 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:53.406453 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:53.407312 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:53.408943 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:53.409358 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:53.409453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:53.412781 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:53.412786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:53.412790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:53.412823 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:53.415632 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:53.416341 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:53.416345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:53.416347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:53.416466 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:53.416907 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:53.416912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:53.416914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:53.417261 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:53.417445 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:53.418323 (+ 878us) server_negotiation.cc:299] Negotiation successful
0807 02:55:53.418575 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":239,"threads_started":1}
I20260807 02:55:53.419936 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:53.400729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37329 (local address 127.1.121.67:46577)
0807 02:55:53.401521 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:53.401558 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:53.403903 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:53.404676 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:53.404685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:53.405395 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:53.406249 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:53.406275 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:53.407466 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:53.407470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:53.408750 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:53.408762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:53.409563 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:53.409568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:53.409783 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:53.410421 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:53.410443 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:53.412562 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:53.415781 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:53.415788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:53.415801 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:53.416226 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:53.416561 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:53.416564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:53.416566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:53.416757 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:53.417418 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:53.417425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:53.417907 (+ 482us) client_negotiation.cc:769] Sending connection context
0807 02:55:53.418171 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 02:55:53.418730 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":284,"threads_started":1}
I20260807 02:55:53.421332 5700 heartbeater.cc:347] Connected to a master server at 127.1.121.126:37329
I20260807 02:55:53.421680 5700 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:53.422430 5700 heartbeater.cc:511] Master 127.1.121.126:37329 requested a full tablet report, sending...
I20260807 02:55:53.423882 5054 ts_manager.cc:194] Registered new tserver with Master: ec8aedd23b05490397ecce81d70a658a (127.1.121.67:43819)
I20260807 02:55:53.424736 5054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:46577
I20260807 02:55:53.431444 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 02:55:53.472271 5703 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:55:53.469589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.1.121.126:37329)
0807 02:55:53.469777 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:53.469782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:53.470331 (+ 549us) server_negotiation.cc:407] Connection header received
0807 02:55:53.470493 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:53.470497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:53.470582 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:53.470679 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:53.471935 (+ 1256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37398: BlockingRecv error: recv got EOF from 127.0.0.1:37398 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:55:53.475803 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 5148
W20260807 02:55:53.478694 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 5148: Not authorized: Client connection negotiation failed: client connection to 127.1.121.65:38525: server requires authentication, but client does not have Kerberos credentials available
I20260807 02:55:53.478885 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 5148
I20260807 02:55:53.502272 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 5341
W20260807 02:55:53.505168 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 5341: Not authorized: Client connection negotiation failed: client connection to 127.1.121.66:38015: server requires authentication, but client does not have Kerberos credentials available
I20260807 02:55:53.505350 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 5341
I20260807 02:55:53.530754 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 5535
W20260807 02:55:53.533993 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 5535: Not authorized: Client connection negotiation failed: client connection to 127.1.121.67:43819: server requires authentication, but client does not have Kerberos credentials available
I20260807 02:55:53.534191 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 5535
I20260807 02:55:53.559226 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 5010
I20260807 02:55:53.560748 5099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:53.670979 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 5010
2026-08-07T02:55:53Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4039 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:55:53 dist-test-slave-3h5h krb5kdc[5725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:55:53 dist-test-slave-3h5h krb5kdc[5725](info): set up 2 sockets
Aug 07 02:55:53 dist-test-slave-3h5h krb5kdc[5725](info): commencing operation
krb5kdc: starting...
W20260807 02:55:54.985651 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:55:55 dist-test-slave-3h5h krb5kdc[5725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:55:55.241304 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33745
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:33745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:55.609090 5742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:55.609408 5742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:55.609587 5742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:55.617295 5742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:55:55.617406 5742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:55.617519 5742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:55.617626 5742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:55:55.617718 5742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:55:55.627993 5742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:33745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:55.629365 5742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:55.630714 5742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:55.642980 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:55.643054 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:55.643075 5752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:55.644325 5742 server_base.cc:1034] running on GCE node
I20260807 02:55:55.645356 5742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:55.646864 5742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:55.648128 5742 hybrid_clock.cc:648] HybridClock initialized: now 1786071355648078 us; error 29 us; skew 500 ppm
I20260807 02:55:55.648545 5742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:55 dist-test-slave-3h5h krb5kdc[5725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071355, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:55.651968 5742 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:55:55.653512 5742 webserver.cc:460] Webserver started at http://127.1.121.126:43841/ using document root <none> and password file <none>
I20260807 02:55:55.654286 5742 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:55.654423 5742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:55.654779 5742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:55.657918 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "0eebbb787e434532b930f4a465125c6e"
format_stamp: "Formatted at 2026-08-07 02:55:55 on dist-test-slave-3h5h"
server_key: "9c2df1b4607366cee6dabb6c5bd04a35"
server_key_iv: "7f167cf2826dbd9f4655612bd4ddb2de"
server_key_version: "encryptionkey@0"
I20260807 02:55:55.658769 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "0eebbb787e434532b930f4a465125c6e"
format_stamp: "Formatted at 2026-08-07 02:55:55 on dist-test-slave-3h5h"
server_key: "9c2df1b4607366cee6dabb6c5bd04a35"
server_key_iv: "7f167cf2826dbd9f4655612bd4ddb2de"
server_key_version: "encryptionkey@0"
I20260807 02:55:55.663987 5742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 02:55:55.667582 5761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:55.669082 5742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 02:55:55.669282 5742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "0eebbb787e434532b930f4a465125c6e"
format_stamp: "Formatted at 2026-08-07 02:55:55 on dist-test-slave-3h5h"
server_key: "9c2df1b4607366cee6dabb6c5bd04a35"
server_key_iv: "7f167cf2826dbd9f4655612bd4ddb2de"
server_key_version: "encryptionkey@0"
I20260807 02:55:55.669539 5742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:55.682485 5742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:55.687165 5742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:55.687494 5742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:55.707926 5742 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:33745
I20260807 02:55:55.707927 5831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:33745 every 8 connection(s)
I20260807 02:55:55.709671 5742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:55:55.713547 5833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:55.715437 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 5742
I20260807 02:55:55.715672 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:55:55.716490 1509 external_mini_cluster.cc:1428] Setting key b607db9e4a594ce4ccf0914671fa601f
I20260807 02:55:55.726799 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e: Bootstrap starting.
I20260807 02:55:55.730942 5833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e: Neither blocks nor log segments found. Creating new log.
Aug 07 02:55:55 dist-test-slave-3h5h krb5kdc[5725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:55.732193 5833 log.cc:824] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:55.735167 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e: No bootstrap required, opened a new log
I20260807 02:55:55.740760 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:55.718533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.1.121.126:33745)
0807 02:55:55.719401 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:55.719413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:55.719461 (+ 48us) server_negotiation.cc:407] Connection header received
0807 02:55:55.721355 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:55.721378 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:55.721774 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:55.722096 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:55.723970 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:55.725406 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:55.727206 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:55.727931 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:55.728072 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:55.732988 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:55.732999 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:55.733015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:55.733078 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:55.736279 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:55.737116 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:55.737123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:55.737130 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:55.737303 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:55.737729 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:55.737745 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:55.737747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:55.738318 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:55.738665 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:55.739134 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 02:55:55.739627 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":406,"threads_started":1}
I20260807 02:55:55.743911 5833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eebbb787e434532b930f4a465125c6e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33745 } }
I20260807 02:55:55.744412 5833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:55.744524 5833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eebbb787e434532b930f4a465125c6e, State: Initialized, Role: FOLLOWER
I20260807 02:55:55.745384 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:55:55.745304 5833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eebbb787e434532b930f4a465125c6e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33745 } }
I20260807 02:55:55.745535 5833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:55:55.745672 5833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:55:55.745841 5833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:55.748034 5833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eebbb787e434532b930f4a465125c6e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33745 } }
I20260807 02:55:55.748560 5833 leader_election.cc:302] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eebbb787e434532b930f4a465125c6e; no voters:
I20260807 02:55:55.749289 5833 leader_election.cc:288] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:55:55.749395 5839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:55.751986 5833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:55:55.756062 5839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [term 1 LEADER]: Becoming Leader. State: Replica: 0eebbb787e434532b930f4a465125c6e, State: Running, Role: LEADER
I20260807 02:55:55.756757 5839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eebbb787e434532b930f4a465125c6e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33745 } }
I20260807 02:55:55.760413 5840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eebbb787e434532b930f4a465125c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eebbb787e434532b930f4a465125c6e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33745 } } }
I20260807 02:55:55.760919 5840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:55.761408 5844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eebbb787e434532b930f4a465125c6e. Latest consensus state: current_term: 1 leader_uuid: "0eebbb787e434532b930f4a465125c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eebbb787e434532b930f4a465125c6e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33745 } } }
I20260807 02:55:55.761756 5844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:55.763876 5856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:55:55.770432 5856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:55:55.782788 5856 catalog_manager.cc:1347] Generated new cluster ID: cdd01cb2b80f40ada54a0bfbb354458b
I20260807 02:55:55.782920 5856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:55:55.796710 5856 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:55:55.798717 5856 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:55:55.805723 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:55.794553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.1.121.126:33745)
0807 02:55:55.795038 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:55.795045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:55.796096 (+ 1051us) server_negotiation.cc:407] Connection header received
0807 02:55:55.796300 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:55.796306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:55.796393 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:55.796552 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:55.798285 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:55.799411 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:55.801401 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:55.802199 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:55.802332 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:55.803668 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:55.803675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:55.803681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:55:55.804763 (+ 1082us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:55.804871 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:55:55.804909 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:55.805031 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:55.805194 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 02:55:55.805341 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":322}
I20260807 02:55:55.809923 5856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eebbb787e434532b930f4a465125c6e: Generated new TSK 0
I20260807 02:55:55.810904 5856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:55:55.824528 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 5742
I20260807 02:55:55.825794 5826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:55.916651 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 5742
2026-08-07T02:55:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2237 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:55:56 dist-test-slave-3h5h krb5kdc[5878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:55:56 dist-test-slave-3h5h krb5kdc[5878](info): set up 2 sockets
Aug 07 02:55:56 dist-test-slave-3h5h krb5kdc[5878](info): commencing operation
krb5kdc: starting...
W20260807 02:55:57.228915 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:55:57 dist-test-slave-3h5h krb5kdc[5878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:55:57.470839 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:42209
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:42209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:57.805521 5895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:57.805893 5895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:57.806073 5895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:57.813256 5895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:55:57.813416 5895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:57.813529 5895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:57.813637 5895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:55:57.813742 5895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:55:57.823590 5895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:42209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:57.825028 5895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:57.826396 5895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:57.838810 5901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:57.838912 5904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:57.838868 5902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:57.839468 5895 server_base.cc:1034] running on GCE node
I20260807 02:55:57.840365 5895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:57.841966 5895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:57.843230 5895 hybrid_clock.cc:648] HybridClock initialized: now 1786071357843172 us; error 37 us; skew 500 ppm
I20260807 02:55:57.843631 5895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:57 dist-test-slave-3h5h krb5kdc[5878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071357, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:57.851289 5895 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:55:57.852897 5895 webserver.cc:460] Webserver started at http://127.1.121.126:34021/ using document root <none> and password file <none>
I20260807 02:55:57.853642 5895 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:57.853807 5895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:57.854140 5895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:57.857306 5895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "db2501d79e4f47fc9fc05947d02934e7"
format_stamp: "Formatted at 2026-08-07 02:55:57 on dist-test-slave-3h5h"
server_key: "4aa8852b120b79860bdef69ce97980c1"
server_key_iv: "ea1eb0337cbe7865546d989e3a5b98ee"
server_key_version: "encryptionkey@0"
I20260807 02:55:57.858109 5895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "db2501d79e4f47fc9fc05947d02934e7"
format_stamp: "Formatted at 2026-08-07 02:55:57 on dist-test-slave-3h5h"
server_key: "4aa8852b120b79860bdef69ce97980c1"
server_key_iv: "ea1eb0337cbe7865546d989e3a5b98ee"
server_key_version: "encryptionkey@0"
I20260807 02:55:57.864151 5895 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 02:55:57.868360 5915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:57.869817 5895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:55:57.870025 5895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "db2501d79e4f47fc9fc05947d02934e7"
format_stamp: "Formatted at 2026-08-07 02:55:57 on dist-test-slave-3h5h"
server_key: "4aa8852b120b79860bdef69ce97980c1"
server_key_iv: "ea1eb0337cbe7865546d989e3a5b98ee"
server_key_version: "encryptionkey@0"
I20260807 02:55:57.870255 5895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:57.896998 5895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:57.901552 5895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:57.901937 5895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:57.922211 5895 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:42209
I20260807 02:55:57.922253 5983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:42209 every 8 connection(s)
I20260807 02:55:57.923812 5895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:55:57.927613 5985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:55:57.934111 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 5895
I20260807 02:55:57.934353 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:55:57.934995 1509 external_mini_cluster.cc:1428] Setting key 6082af01382153ac21f4dcb6c353aaeb
I20260807 02:55:57.938776 5985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7: Bootstrap starting.
I20260807 02:55:57.941952 5985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7: Neither blocks nor log segments found. Creating new log.
I20260807 02:55:57.943085 5985 log.cc:824] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7: Log is configured to *not* fsync() on all Append() calls
I20260807 02:55:57.946118 5985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7: No bootstrap required, opened a new log
Aug 07 02:55:57 dist-test-slave-3h5h krb5kdc[5878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:57.954207 5985 raft_consensus.cc:348] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2501d79e4f47fc9fc05947d02934e7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42209 } }
I20260807 02:55:57.954666 5985 raft_consensus.cc:374] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:55:57.954795 5985 raft_consensus.cc:729] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2501d79e4f47fc9fc05947d02934e7, State: Initialized, Role: FOLLOWER
I20260807 02:55:57.955530 5985 consensus_queue.cc:260] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2501d79e4f47fc9fc05947d02934e7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42209 } }
I20260807 02:55:57.955822 5985 raft_consensus.cc:388] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:55:57.955979 5985 raft_consensus.cc:482] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:55:57.956180 5985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:55:57.958817 5985 raft_consensus.cc:504] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2501d79e4f47fc9fc05947d02934e7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42209 } }
I20260807 02:55:57.959322 5988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:57.937309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.1.121.126:42209)
0807 02:55:57.938162 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:57.938174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:57.938218 (+ 44us) server_negotiation.cc:407] Connection header received
0807 02:55:57.940019 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:57.940052 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:57.940442 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:57.940824 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:57.942416 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:57.944702 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:57.946136 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:57.946918 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:57.947154 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:57.951357 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:57.951368 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:57.951385 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:57.951421 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:57.954705 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:57.955606 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:57.955613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:57.955619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:57.955857 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:57.956330 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:57.956336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:57.956340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:57.956998 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:57.957356 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:57.957771 (+ 415us) server_negotiation.cc:299] Negotiation successful
0807 02:55:57.958236 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":384,"threads_started":1}
I20260807 02:55:57.959359 5985 leader_election.cc:302] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db2501d79e4f47fc9fc05947d02934e7; no voters:
I20260807 02:55:57.960244 5985 leader_election.cc:288] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:55:57.960455 5990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:55:57.962920 5985 sys_catalog.cc:564] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:55:57.963016 5990 raft_consensus.cc:686] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [term 1 LEADER]: Becoming Leader. State: Replica: db2501d79e4f47fc9fc05947d02934e7, State: Running, Role: LEADER
I20260807 02:55:57.963724 5990 consensus_queue.cc:237] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2501d79e4f47fc9fc05947d02934e7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42209 } }
I20260807 02:55:57.967099 5992 sys_catalog.cc:455] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db2501d79e4f47fc9fc05947d02934e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2501d79e4f47fc9fc05947d02934e7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42209 } } }
I20260807 02:55:57.967553 5992 sys_catalog.cc:458] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:57.968143 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db2501d79e4f47fc9fc05947d02934e7. Latest consensus state: current_term: 1 leader_uuid: "db2501d79e4f47fc9fc05947d02934e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2501d79e4f47fc9fc05947d02934e7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42209 } } }
I20260807 02:55:57.968505 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7 [sys.catalog]: This master's current role is: LEADER
I20260807 02:55:57.973500 6005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:55:57.980489 6005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:55:57.990089 6005 catalog_manager.cc:1347] Generated new cluster ID: e5011c701f6140d28c65662129313e38
I20260807 02:55:57.990232 6005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:55:58.020320 6005 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:55:58.021706 6005 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:55:58.033524 6005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db2501d79e4f47fc9fc05947d02934e7: Generated new TSK 0
I20260807 02:55:58.034672 6005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:55:58.097450 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42209
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:58.445374 6020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:58.445699 6020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:58.445875 6020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:58.453282 6020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:58.453434 6020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:58.453619 6020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:55:58.463765 6020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:58.465268 6020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:58.466799 6020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:58.480643 6033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:58.480686 6029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:58.480896 6026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:58.481354 6020 server_base.cc:1034] running on GCE node
I20260807 02:55:58.481987 6020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:58.483049 6020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:58.484292 6020 hybrid_clock.cc:648] HybridClock initialized: now 1786071358484239 us; error 43 us; skew 500 ppm
I20260807 02:55:58.484678 6020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:58 dist-test-slave-3h5h krb5kdc[5878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:58.488252 6020 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:55:58.489778 6020 webserver.cc:460] Webserver started at http://127.1.121.65:41915/ using document root <none> and password file <none>
I20260807 02:55:58.490487 6020 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:58.490638 6020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:58.490963 6020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:58.493992 6020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "819790db16ff4520a3b25ad14dc05b07"
format_stamp: "Formatted at 2026-08-07 02:55:58 on dist-test-slave-3h5h"
server_key: "0ef90439663a52180a4560261de182cb"
server_key_iv: "6a0ab303f63dd0022d0a1402d8a5be57"
server_key_version: "encryptionkey@0"
I20260807 02:55:58.494772 6020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "819790db16ff4520a3b25ad14dc05b07"
format_stamp: "Formatted at 2026-08-07 02:55:58 on dist-test-slave-3h5h"
server_key: "0ef90439663a52180a4560261de182cb"
server_key_iv: "6a0ab303f63dd0022d0a1402d8a5be57"
server_key_version: "encryptionkey@0"
I20260807 02:55:58.499964 6020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 02:55:58.503630 6041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:58.505141 6020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:55:58.505338 6020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "819790db16ff4520a3b25ad14dc05b07"
format_stamp: "Formatted at 2026-08-07 02:55:58 on dist-test-slave-3h5h"
server_key: "0ef90439663a52180a4560261de182cb"
server_key_iv: "6a0ab303f63dd0022d0a1402d8a5be57"
server_key_version: "encryptionkey@0"
I20260807 02:55:58.505551 6020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:58.521142 6020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:58.525539 6020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:58.525852 6020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:58.526880 6020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:58.528213 6020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:58.528318 6020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:58.528472 6020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:58.528546 6020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:58.564466 6020 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:42733
I20260807 02:55:58.564517 6198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:42733 every 8 connection(s)
I20260807 02:55:58.566011 6020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:55:58.573184 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 6020
I20260807 02:55:58.573467 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:55:58.574260 1509 external_mini_cluster.cc:1428] Setting key 24d32e134c107832206f4a0c37cba8e1
Aug 07 02:55:58 dist-test-slave-3h5h krb5kdc[5878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:58.589257 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:58.568720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:42369 (local address 127.1.121.126:42209)
0807 02:55:58.569222 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:58.569227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:58.570107 (+ 880us) server_negotiation.cc:407] Connection header received
0807 02:55:58.571975 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:58.571982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:58.572081 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:58.572211 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:58.573955 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:58.574847 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:58.576740 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:58.577204 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:58.577339 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:58.581562 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:58.581573 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:58.581581 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:58.581633 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:58.585507 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:58.586571 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:58.586576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:58.586578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:58.586730 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:58.587283 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:58.587287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:58.587289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:58.587594 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:58.587720 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:58.588691 (+ 971us) server_negotiation.cc:299] Negotiation successful
0807 02:55:58.588977 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
I20260807 02:55:58.590840 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:58.569215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42209 (local address 127.1.121.65:42369)
0807 02:55:58.569938 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:58.569977 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:58.571683 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:58.572462 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:58.572470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:58.573071 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:58.573779 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:58.573800 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:58.575013 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:58.575018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:58.576543 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:58.576556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:58.577449 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:58.577457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:58.577700 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:58.578396 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:58.578419 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:58.581285 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:58.585672 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:58.585685 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:58.585706 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:58.586412 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:58.586851 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:58.586859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:58.586866 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:58.587126 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:58.587733 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:58.587745 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:58.588238 (+ 493us) client_negotiation.cc:769] Sending connection context
0807 02:55:58.588547 (+ 309us) client_negotiation.cc:240] Negotiation successful
0807 02:55:58.589211 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":277,"threads_started":1}
I20260807 02:55:58.592679 6200 heartbeater.cc:347] Connected to a master server at 127.1.121.126:42209
I20260807 02:55:58.593092 6200 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:58.593909 6200 heartbeater.cc:511] Master 127.1.121.126:42209 requested a full tablet report, sending...
I20260807 02:55:58.596489 5938 ts_manager.cc:194] Registered new tserver with Master: 819790db16ff4520a3b25ad14dc05b07 (127.1.121.65:42733)
I20260807 02:55:58.598793 5938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:42369
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:55:58.631989 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42209
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:58.968103 6210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:58.968412 6210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:58.968605 6210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:58.976225 6210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:58.976339 6210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:58.976536 6210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:55:58.986326 6210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:58.987684 6210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:58.989022 6210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:59.002485 6216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:59.002702 6217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:59.003110 6219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:59.003361 6210 server_base.cc:1034] running on GCE node
I20260807 02:55:59.004271 6210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:59.005506 6210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:59.006773 6210 hybrid_clock.cc:648] HybridClock initialized: now 1786071359006710 us; error 52 us; skew 500 ppm
I20260807 02:55:59.007170 6210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:59 dist-test-slave-3h5h krb5kdc[5878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:59.010978 6210 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:55:59.012596 6210 webserver.cc:460] Webserver started at http://127.1.121.66:35827/ using document root <none> and password file <none>
I20260807 02:55:59.013382 6210 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:59.013545 6210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:59.013888 6210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:59.017277 6210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "7f1ab7c150e94d06a85919e079b69639"
format_stamp: "Formatted at 2026-08-07 02:55:59 on dist-test-slave-3h5h"
server_key: "206b1318df90ba4e8c36a8707dbbc44e"
server_key_iv: "b5c3df2ef8879ba9efb5592eafa15258"
server_key_version: "encryptionkey@0"
I20260807 02:55:59.018195 6210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "7f1ab7c150e94d06a85919e079b69639"
format_stamp: "Formatted at 2026-08-07 02:55:59 on dist-test-slave-3h5h"
server_key: "206b1318df90ba4e8c36a8707dbbc44e"
server_key_iv: "b5c3df2ef8879ba9efb5592eafa15258"
server_key_version: "encryptionkey@0"
I20260807 02:55:59.023686 6210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:55:59.027457 6231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:59.029134 6210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:55:59.029321 6210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "7f1ab7c150e94d06a85919e079b69639"
format_stamp: "Formatted at 2026-08-07 02:55:59 on dist-test-slave-3h5h"
server_key: "206b1318df90ba4e8c36a8707dbbc44e"
server_key_iv: "b5c3df2ef8879ba9efb5592eafa15258"
server_key_version: "encryptionkey@0"
I20260807 02:55:59.029531 6210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:59.063320 6210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:59.068115 6210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:59.068427 6210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:59.069449 6210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:59.070914 6210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:59.071027 6210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:59.071192 6210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:59.071267 6210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:59.106343 6210 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:44607
I20260807 02:55:59.106385 6385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:44607 every 8 connection(s)
I20260807 02:55:59.107678 6210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:55:59.116407 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 6210
I20260807 02:55:59.116662 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:55:59.117237 1509 external_mini_cluster.cc:1428] Setting key 0a413932f5ba9064a61c825a5791ee64
Aug 07 02:55:59 dist-test-slave-3h5h krb5kdc[5878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:59.129619 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:59.110446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:54967 (local address 127.1.121.126:42209)
0807 02:55:59.110920 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:59.110924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:59.111763 (+ 839us) server_negotiation.cc:407] Connection header received
0807 02:55:59.113731 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:59.113736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:59.113824 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:59.113949 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:59.115428 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.116173 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.118110 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.118691 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.118797 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:59.122526 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:59.122536 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:59.122544 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:59.122593 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:59.126265 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.127062 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.127070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.127077 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.127240 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.127647 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.127654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.127660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.128078 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:59.128256 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:59.129140 (+ 884us) server_negotiation.cc:299] Negotiation successful
0807 02:55:59.129314 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":266,"threads_started":1}
I20260807 02:55:59.131127 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:59.110906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42209 (local address 127.1.121.66:54967)
0807 02:55:59.111591 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:59.111633 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:59.113438 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:59.114198 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:59.114205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:59.114698 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:59.115254 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:59.115271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.116343 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.116348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:59.117877 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:59.117893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.118938 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.118945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:59.119266 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:59.120017 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:59.120043 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:59.122234 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:59.126486 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:59.126493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:59.126507 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:59.126925 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:59.127350 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:59.127353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:59.127355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:59.127528 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:59.128208 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:59.128214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:59.128684 (+ 470us) client_negotiation.cc:769] Sending connection context
0807 02:55:59.129032 (+ 348us) client_negotiation.cc:240] Negotiation successful
0807 02:55:59.129639 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":283,"threads_started":1}
I20260807 02:55:59.132721 6386 heartbeater.cc:347] Connected to a master server at 127.1.121.126:42209
I20260807 02:55:59.133105 6386 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:59.134063 6386 heartbeater.cc:511] Master 127.1.121.126:42209 requested a full tablet report, sending...
I20260807 02:55:59.135584 5938 ts_manager.cc:194] Registered new tserver with Master: 7f1ab7c150e94d06a85919e079b69639 (127.1.121.66:44607)
I20260807 02:55:59.136405 5938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:54967
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:55:59.175912 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42209
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:55:59.519953 6393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:55:59.520238 6393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:55:59.520334 6393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:55:59.527909 6393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:55:59.528018 6393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:55:59.528132 6393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:55:59.538631 6393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:55:59.539950 6393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:55:59.541301 6393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:55:59.555105 6400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:59.555553 6404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:55:59.555650 6402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:55:59.556048 6393 server_base.cc:1034] running on GCE node
I20260807 02:55:59.556672 6393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:55:59.557789 6393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:55:59.558988 6393 hybrid_clock.cc:648] HybridClock initialized: now 1786071359558939 us; error 46 us; skew 500 ppm
I20260807 02:55:59.559319 6393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:55:59 dist-test-slave-3h5h krb5kdc[5878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:55:59.562608 6393 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:55:59.564317 6393 webserver.cc:460] Webserver started at http://127.1.121.67:38767/ using document root <none> and password file <none>
I20260807 02:55:59.565128 6393 fs_manager.cc:359] Metadata directory not provided
I20260807 02:55:59.565289 6393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:55:59.565662 6393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:55:59.568804 6393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "0344886698dd4508a76005ed574455e3"
format_stamp: "Formatted at 2026-08-07 02:55:59 on dist-test-slave-3h5h"
server_key: "728ad5d7f977042c86228c9679a7062f"
server_key_iv: "16ed32d6e51bffccc3979bfc146c9a81"
server_key_version: "encryptionkey@0"
I20260807 02:55:59.569666 6393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "0344886698dd4508a76005ed574455e3"
format_stamp: "Formatted at 2026-08-07 02:55:59 on dist-test-slave-3h5h"
server_key: "728ad5d7f977042c86228c9679a7062f"
server_key_iv: "16ed32d6e51bffccc3979bfc146c9a81"
server_key_version: "encryptionkey@0"
I20260807 02:55:59.574914 6393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:55:59.578523 6413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:59.579852 6393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:55:59.580039 6393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "0344886698dd4508a76005ed574455e3"
format_stamp: "Formatted at 2026-08-07 02:55:59 on dist-test-slave-3h5h"
server_key: "728ad5d7f977042c86228c9679a7062f"
server_key_iv: "16ed32d6e51bffccc3979bfc146c9a81"
server_key_version: "encryptionkey@0"
I20260807 02:55:59.580245 6393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:55:59.602267 6200 heartbeater.cc:503] Master 127.1.121.126:42209 was elected leader, sending a full tablet report...
I20260807 02:55:59.603989 6393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:55:59.608714 6393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:55:59.609062 6393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:55:59.610226 6393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:55:59.611685 6393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:55:59.611763 6393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:59.611943 6393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:55:59.611995 6393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:55:59.648754 6393 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:43409
I20260807 02:55:59.648803 6574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:43409 every 8 connection(s)
I20260807 02:55:59.650125 6393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:55:59.659855 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 6393
I20260807 02:55:59.660075 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:55:59.660578 1509 external_mini_cluster.cc:1428] Setting key 58a0fffdd35d2e06ac08a6bc538d2c05
Aug 07 02:55:59 dist-test-slave-3h5h krb5kdc[5878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:59.671998 6577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:59.652923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:43087 (local address 127.1.121.126:42209)
0807 02:55:59.653456 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:59.653463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:59.654427 (+ 964us) server_negotiation.cc:407] Connection header received
0807 02:55:59.657039 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:59.657054 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:59.657158 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:59.657288 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:59.658905 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.659970 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.661913 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.662410 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.662577 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:59.666501 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:59.666506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:59.666510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:59.666542 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:59.669287 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.669994 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.669998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.670000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.670125 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.670526 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.670529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.670531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.670806 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:59.670916 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:59.671557 (+ 641us) server_negotiation.cc:299] Negotiation successful
0807 02:55:59.671732 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":265,"threads_started":1}
I20260807 02:55:59.673063 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:59.653448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42209 (local address 127.1.121.67:43087)
0807 02:55:59.654263 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 02:55:59.654301 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 02:55:59.656646 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:55:59.657561 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:55:59.657569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:55:59.658142 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:55:59.658722 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:59.658740 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.660160 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.660164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:59.661753 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:55:59.661767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.662698 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.662708 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:55:59.662995 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:59.663773 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:55:59.663801 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:55:59.666267 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:55:59.669413 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:59.669420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:59.669433 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:59.669881 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:59.670223 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:55:59.670226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:55:59.670228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:55:59.670415 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:55:59.670915 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:55:59.670921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:55:59.671230 (+ 309us) client_negotiation.cc:769] Sending connection context
0807 02:55:59.671452 (+ 222us) client_negotiation.cc:240] Negotiation successful
0807 02:55:59.671849 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":321,"threads_started":1}
I20260807 02:55:59.674664 6575 heartbeater.cc:347] Connected to a master server at 127.1.121.126:42209
I20260807 02:55:59.675061 6575 heartbeater.cc:464] Registering TS with master...
I20260807 02:55:59.676093 6575 heartbeater.cc:511] Master 127.1.121.126:42209 requested a full tablet report, sending...
I20260807 02:55:59.677628 5938 ts_manager.cc:194] Registered new tserver with Master: 0344886698dd4508a76005ed574455e3 (127.1.121.67:43409)
I20260807 02:55:59.678399 5938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:43087
I20260807 02:55:59.688524 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:55:59 dist-test-slave-3h5h krb5kdc[5878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071359, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 02:55:59 dist-test-slave-3h5h krb5kdc[5878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071359, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:55:59.747421 6577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:59.733097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.1.121.126:42209)
0807 02:55:59.733296 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:59.733300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:59.733842 (+ 542us) server_negotiation.cc:407] Connection header received
0807 02:55:59.733971 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:59.733976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:59.734069 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:59.734169 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:59.735598 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.736426 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.737775 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.738224 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.738328 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:59.742205 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:59.742209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:59.742213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:59.742248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:59.744707 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.745410 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.745415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.745418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.745619 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.746057 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.746061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.746063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.746421 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:59.746644 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:59.746893 (+ 249us) server_negotiation.cc:299] Negotiation successful
0807 02:55:59.747184 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 02:55:59.748119 5938 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48872
I20260807 02:55:59.750291 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 6020
Aug 07 02:55:59 dist-test-slave-3h5h krb5kdc[5878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071359, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:55:59.766942 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:59.751575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.1.121.65:42733)
0807 02:55:59.752218 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:59.752226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:59.752250 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:55:59.752307 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:59.752314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:59.752519 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:59.752642 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:59.753910 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.754936 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.756142 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.756701 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.756796 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:59.760768 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:59.760779 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:59.760792 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:59.760823 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:59.763648 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.764384 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.764391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.764393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.764554 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.765315 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.765319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.765321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.765722 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:59.765952 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:59.766321 (+ 369us) server_negotiation.cc:299] Negotiation successful
0807 02:55:59.766533 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":329,"threads_started":1}
W20260807 02:55:59.768327 6194 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39466
W20260807 02:55:59.769482 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 6020: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 02:55:59.769727 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 6020
I20260807 02:55:59.797387 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 6210
Aug 07 02:55:59 dist-test-slave-3h5h krb5kdc[5878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071359, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:55:59.814476 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:59.798592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.1.121.66:44607)
0807 02:55:59.799214 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:59.799221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:59.799249 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:55:59.799320 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:59.799327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:59.799548 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:59.799683 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:59.800983 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.802154 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.803398 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.803999 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.804102 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:59.808350 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:59.808365 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:59.808382 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:59.808417 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:59.811808 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.812425 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.812433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.812436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.812596 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.813000 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.813004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.813006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.813365 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:59.813609 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:59.813942 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 02:55:59.814164 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":313,"threads_started":1}
W20260807 02:55:59.815832 6379 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49830
W20260807 02:55:59.816947 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 6210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 02:55:59.817152 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 6210
I20260807 02:55:59.842110 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 6393
Aug 07 02:55:59 dist-test-slave-3h5h krb5kdc[5878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071359, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:55:59.859233 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:55:59.843197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.1.121.67:43409)
0807 02:55:59.843900 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0807 02:55:59.843908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:55:59.843936 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:55:59.843998 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:55:59.844004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:55:59.844219 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:55:59.844348 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:55:59.845773 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.846831 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.848091 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:55:59.848663 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:55:59.848762 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:55:59.853420 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:55:59.853434 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:55:59.853451 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:55:59.853494 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:55:59.856315 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.857276 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.857284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.857287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.857416 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:55:59.857754 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:55:59.857757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:55:59.857759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:55:59.858101 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:55:59.858340 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 02:55:59.858593 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 02:55:59.858819 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":359,"threads_started":1}
W20260807 02:55:59.860620 6569 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34126
W20260807 02:55:59.861754 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 6393: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 02:55:59.861930 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 6393
I20260807 02:55:59.885125 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 5895
I20260807 02:55:59.886363 5979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:55:59.992332 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 5895
2026-08-07T02:56:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4082 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:56:00 dist-test-slave-3h5h krb5kdc[6601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:56:00 dist-test-slave-3h5h krb5kdc[6601](info): set up 2 sockets
Aug 07 02:56:00 dist-test-slave-3h5h krb5kdc[6601](info): commencing operation
krb5kdc: starting...
W20260807 02:56:01.346771 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:56:01 dist-test-slave-3h5h krb5kdc[6601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:56:01.586696 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:42295
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:42295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:01.929149 6622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:01.929469 6622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:01.929669 6622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:01.936995 6622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:01.937112 6622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:01.937239 6622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:01.937342 6622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:01.937430 6622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:01.947516 6622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:42295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:42295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:01.948940 6622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:01.950383 6622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:01.963517 6631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:01.963469 6633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:01.963475 6636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:01.964516 6622 server_base.cc:1034] running on GCE node
I20260807 02:56:01.965394 6622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:01.966832 6622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:01.968250 6622 hybrid_clock.cc:648] HybridClock initialized: now 1786071361968206 us; error 52 us; skew 500 ppm
I20260807 02:56:01.968688 6622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:01 dist-test-slave-3h5h krb5kdc[6601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:01.971971 6622 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:56:01.973554 6622 webserver.cc:460] Webserver started at http://127.1.121.126:36065/ using document root <none> and password file <none>
I20260807 02:56:01.974280 6622 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:01.974442 6622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:01.974777 6622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:01.977896 6622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "2dd15492d4144a57a93e53902e61bfca"
format_stamp: "Formatted at 2026-08-07 02:56:01 on dist-test-slave-3h5h"
server_key: "f4adcfc2e082217a86a667be7ce1bea9"
server_key_iv: "e96a4d8c646ff39228fd6fc47706101c"
server_key_version: "encryptionkey@0"
I20260807 02:56:01.978699 6622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "2dd15492d4144a57a93e53902e61bfca"
format_stamp: "Formatted at 2026-08-07 02:56:01 on dist-test-slave-3h5h"
server_key: "f4adcfc2e082217a86a667be7ce1bea9"
server_key_iv: "e96a4d8c646ff39228fd6fc47706101c"
server_key_version: "encryptionkey@0"
I20260807 02:56:01.983923 6622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 02:56:01.987452 6649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:01.988783 6622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:56:01.989009 6622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "2dd15492d4144a57a93e53902e61bfca"
format_stamp: "Formatted at 2026-08-07 02:56:01 on dist-test-slave-3h5h"
server_key: "f4adcfc2e082217a86a667be7ce1bea9"
server_key_iv: "e96a4d8c646ff39228fd6fc47706101c"
server_key_version: "encryptionkey@0"
I20260807 02:56:01.989216 6622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:02.002486 6622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:02.007107 6622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:02.007431 6622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:02.026672 6622 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:42295
I20260807 02:56:02.026716 6722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:42295 every 8 connection(s)
I20260807 02:56:02.028204 6622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:02.030143 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 6622
I20260807 02:56:02.030395 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:02.031101 1509 external_mini_cluster.cc:1428] Setting key de87e5e8caa80b50ac8c4d9456cb9483
I20260807 02:56:02.032464 6724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 02:56:02 dist-test-slave-3h5h krb5kdc[6601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:02.047385 6724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca: Bootstrap starting.
I20260807 02:56:02.050633 6724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:02.051713 6724 log.cc:824] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:02.054905 6724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca: No bootstrap required, opened a new log
I20260807 02:56:02.056095 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:02.033189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.1.121.126:42295)
0807 02:56:02.034010 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:02.034022 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:02.034064 (+ 42us) server_negotiation.cc:407] Connection header received
0807 02:56:02.035669 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:02.035691 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:02.036077 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:02.036424 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:02.038629 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:02.040041 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:02.041838 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:02.042542 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:02.042680 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:02.047878 (+ 5198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:02.047887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:02.047903 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:02.047945 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:02.051470 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:02.052304 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:02.052316 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:02.052323 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:02.052533 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:02.053026 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:02.053030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:02.053032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:02.053584 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:02.053940 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:02.054544 (+ 604us) server_negotiation.cc:299] Negotiation successful
0807 02:56:02.055051 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":393,"threads_started":1}
I20260807 02:56:02.062896 6724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd15492d4144a57a93e53902e61bfca" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42295 } }
I20260807 02:56:02.063367 6724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:02.063504 6724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd15492d4144a57a93e53902e61bfca, State: Initialized, Role: FOLLOWER
I20260807 02:56:02.064077 6724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd15492d4144a57a93e53902e61bfca" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42295 } }
I20260807 02:56:02.064282 6724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:02.064370 6724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:02.064483 6724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:02.066919 6724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd15492d4144a57a93e53902e61bfca" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42295 } }
I20260807 02:56:02.067464 6724 leader_election.cc:302] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dd15492d4144a57a93e53902e61bfca; no voters:
I20260807 02:56:02.068060 6724 leader_election.cc:288] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:02.068222 6730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:02.068933 6730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [term 1 LEADER]: Becoming Leader. State: Replica: 2dd15492d4144a57a93e53902e61bfca, State: Running, Role: LEADER
I20260807 02:56:02.069461 6730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd15492d4144a57a93e53902e61bfca" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42295 } }
I20260807 02:56:02.070180 6724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:02.072039 6733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dd15492d4144a57a93e53902e61bfca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd15492d4144a57a93e53902e61bfca" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42295 } } }
I20260807 02:56:02.072387 6733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:02.073395 6739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:02.073174 6734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd15492d4144a57a93e53902e61bfca. Latest consensus state: current_term: 1 leader_uuid: "2dd15492d4144a57a93e53902e61bfca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd15492d4144a57a93e53902e61bfca" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42295 } } }
I20260807 02:56:02.074045 6734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:02.079857 6739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:02.089053 6739 catalog_manager.cc:1347] Generated new cluster ID: 17183dc8c0244dc4966f3989934c1221
I20260807 02:56:02.089200 6739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:02.125792 6739 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:02.127748 6739 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:02.142515 6739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dd15492d4144a57a93e53902e61bfca: Generated new TSK 0
I20260807 02:56:02.143463 6739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:56:02.217276 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42295
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:02.569345 6757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:02.569635 6757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:02.569835 6757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:02.577380 6757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:02.577514 6757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:02.577703 6757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:56:02.587440 6757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:02.588774 6757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:02.590147 6757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:02.605798 6764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:02.605922 6763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:02.605893 6766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:02.607172 6757 server_base.cc:1034] running on GCE node
I20260807 02:56:02.607816 6757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:02.608798 6757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:02.610093 6757 hybrid_clock.cc:648] HybridClock initialized: now 1786071362610031 us; error 67 us; skew 500 ppm
I20260807 02:56:02.610428 6757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:02 dist-test-slave-3h5h krb5kdc[6601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:02.613720 6757 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:56:02.615190 6757 webserver.cc:460] Webserver started at http://127.1.121.65:33633/ using document root <none> and password file <none>
I20260807 02:56:02.615836 6757 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:02.615938 6757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:02.616212 6757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:02.619194 6757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "caf16feb77d84f84a58a41080849da86"
format_stamp: "Formatted at 2026-08-07 02:56:02 on dist-test-slave-3h5h"
server_key: "d1c5befee41edbf372d24d623e3f90cc"
server_key_iv: "51e77f410ed80beaf166ab5b02354386"
server_key_version: "encryptionkey@0"
I20260807 02:56:02.619941 6757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "caf16feb77d84f84a58a41080849da86"
format_stamp: "Formatted at 2026-08-07 02:56:02 on dist-test-slave-3h5h"
server_key: "d1c5befee41edbf372d24d623e3f90cc"
server_key_iv: "51e77f410ed80beaf166ab5b02354386"
server_key_version: "encryptionkey@0"
I20260807 02:56:02.625087 6757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 02:56:02.628593 6776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:02.630071 6757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:02.630286 6757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "caf16feb77d84f84a58a41080849da86"
format_stamp: "Formatted at 2026-08-07 02:56:02 on dist-test-slave-3h5h"
server_key: "d1c5befee41edbf372d24d623e3f90cc"
server_key_iv: "51e77f410ed80beaf166ab5b02354386"
server_key_version: "encryptionkey@0"
I20260807 02:56:02.630541 6757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:02.660230 6757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:02.665398 6757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:02.665750 6757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:02.666886 6757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:02.668390 6757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:02.668479 6757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:02.668674 6757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:02.668741 6757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:02.707640 6757 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:38513
I20260807 02:56:02.707706 6939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:38513 every 8 connection(s)
I20260807 02:56:02.709134 6757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:56:02.714030 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 6757
I20260807 02:56:02.714257 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:56:02.714901 1509 external_mini_cluster.cc:1428] Setting key fbef94d4ce34f1d958f867481415bae6
Aug 07 02:56:02 dist-test-slave-3h5h krb5kdc[6601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:02.732401 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:02.712030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:60055 (local address 127.1.121.126:42295)
0807 02:56:02.712581 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:02.712587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:02.713656 (+ 1069us) server_negotiation.cc:407] Connection header received
0807 02:56:02.716095 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:02.716101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:02.716199 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:02.716331 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:02.718011 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:02.718782 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:02.720423 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:02.720941 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:02.721099 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:02.725256 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:02.725265 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:02.725271 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:02.725317 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:02.728975 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:02.729728 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:02.729738 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:02.729744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:02.729897 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:02.730326 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:02.730333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:02.730339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:02.730750 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:02.730908 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:02.731648 (+ 740us) server_negotiation.cc:299] Negotiation successful
0807 02:56:02.731915 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":253,"threads_started":1}
I20260807 02:56:02.733053 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:02.712604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42295 (local address 127.1.121.65:60055)
0807 02:56:02.713455 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:02.713521 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:02.715744 (+ 2223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:02.716592 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:02.716601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:02.717231 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:02.717819 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:02.717838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:02.718962 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:02.718967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:02.720247 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:02.720262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:02.721198 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:02.721208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:02.721471 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:02.722230 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:02.722254 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:02.725016 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:02.729117 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:02.729124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:02.729137 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:02.729586 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:02.730008 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:02.730012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:02.730014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:02.730206 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:02.730877 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:02.730882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:02.731223 (+ 341us) client_negotiation.cc:769] Sending connection context
0807 02:56:02.731462 (+ 239us) client_negotiation.cc:240] Negotiation successful
0807 02:56:02.731857 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":355,"threads_started":1}
I20260807 02:56:02.734789 6940 heartbeater.cc:347] Connected to a master server at 127.1.121.126:42295
I20260807 02:56:02.735159 6940 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:02.736162 6940 heartbeater.cc:511] Master 127.1.121.126:42295 requested a full tablet report, sending...
I20260807 02:56:02.738440 6670 ts_manager.cc:194] Registered new tserver with Master: caf16feb77d84f84a58a41080849da86 (127.1.121.65:38513)
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
I20260807 02:56:02.739892 6670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:60055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:56:02.773382 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42295
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:03.136256 6946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:03.136642 6946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:03.136874 6946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:03.145363 6946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:03.145529 6946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:03.145720 6946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:56:03.155967 6946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:03.157598 6946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:03.159026 6946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:03.173161 6953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:03.173254 6955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:03.173439 6960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:03.174537 6946 server_base.cc:1034] running on GCE node
I20260807 02:56:03.175201 6946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:03.176436 6946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:03.177649 6946 hybrid_clock.cc:648] HybridClock initialized: now 1786071363177571 us; error 73 us; skew 500 ppm
I20260807 02:56:03.178004 6946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:03 dist-test-slave-3h5h krb5kdc[6601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:03.186543 6946 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:56:03.188285 6946 webserver.cc:460] Webserver started at http://127.1.121.66:41321/ using document root <none> and password file <none>
I20260807 02:56:03.189160 6946 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:03.189277 6946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:03.189656 6946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:03.193184 6946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "a2465a1580324d74a598bcd5536102c2"
format_stamp: "Formatted at 2026-08-07 02:56:03 on dist-test-slave-3h5h"
server_key: "06be8aa33fea5692f90fd9e4c7bec289"
server_key_iv: "9f0a4d86370d8b1d9d24aa404b531e31"
server_key_version: "encryptionkey@0"
I20260807 02:56:03.194063 6946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "a2465a1580324d74a598bcd5536102c2"
format_stamp: "Formatted at 2026-08-07 02:56:03 on dist-test-slave-3h5h"
server_key: "06be8aa33fea5692f90fd9e4c7bec289"
server_key_iv: "9f0a4d86370d8b1d9d24aa404b531e31"
server_key_version: "encryptionkey@0"
I20260807 02:56:03.199613 6946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:56:03.203240 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:03.204728 6946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 02:56:03.204957 6946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "a2465a1580324d74a598bcd5536102c2"
format_stamp: "Formatted at 2026-08-07 02:56:03 on dist-test-slave-3h5h"
server_key: "06be8aa33fea5692f90fd9e4c7bec289"
server_key_iv: "9f0a4d86370d8b1d9d24aa404b531e31"
server_key_version: "encryptionkey@0"
I20260807 02:56:03.205212 6946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:03.255187 6946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:03.259718 6946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:03.260044 6946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:03.261144 6946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:03.262576 6946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:03.262682 6946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:03.262835 6946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:03.262907 6946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:03.301394 6946 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:33427
I20260807 02:56:03.301470 7121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:33427 every 8 connection(s)
I20260807 02:56:03.303089 6946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:56:03.309158 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 6946
I20260807 02:56:03.309365 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:56:03.309916 1509 external_mini_cluster.cc:1428] Setting key 2c94a08915c07cb8d325f3ceed94e8a3
Aug 07 02:56:03 dist-test-slave-3h5h krb5kdc[6601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:03.325234 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:03.305838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:44275 (local address 127.1.121.126:42295)
0807 02:56:03.306337 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:03.306342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:03.307199 (+ 857us) server_negotiation.cc:407] Connection header received
0807 02:56:03.309671 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:03.309677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:03.309767 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:03.309879 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:03.311694 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.312788 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.314352 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.315117 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.315264 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:03.318693 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:03.318701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:03.318709 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:03.318755 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:03.321865 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.322805 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.322808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.322810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.322931 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.323406 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.323409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.323411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.323663 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:03.323771 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:03.324716 (+ 945us) server_negotiation.cc:299] Negotiation successful
0807 02:56:03.324962 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
I20260807 02:56:03.326841 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:03.306327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42295 (local address 127.1.121.66:44275)
0807 02:56:03.307026 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:03.307063 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:03.309221 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:03.310160 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:03.310170 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:03.310850 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:03.311414 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:03.311431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.312964 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.312967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:03.314200 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:03.314214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.315327 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.315332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:03.315527 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:03.316236 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:03.316264 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:03.318457 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:03.322010 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:03.322022 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:03.322040 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:03.322687 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:03.323038 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:03.323045 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:03.323052 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:03.323299 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:03.323792 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:03.323802 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:03.324267 (+ 465us) client_negotiation.cc:769] Sending connection context
0807 02:56:03.324670 (+ 403us) client_negotiation.cc:240] Negotiation successful
0807 02:56:03.325281 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":283,"threads_started":1}
I20260807 02:56:03.328378 7123 heartbeater.cc:347] Connected to a master server at 127.1.121.126:42295
I20260807 02:56:03.328689 7123 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:03.329492 7123 heartbeater.cc:511] Master 127.1.121.126:42295 requested a full tablet report, sending...
I20260807 02:56:03.331461 6670 ts_manager.cc:194] Registered new tserver with Master: a2465a1580324d74a598bcd5536102c2 (127.1.121.66:33427)
I20260807 02:56:03.332418 6670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:44275
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:56:03.366568 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42295
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:03.710276 7132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:03.710575 7132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:03.710747 7132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:03.718077 7132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:03.718232 7132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:03.718410 7132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:56:03.728217 7132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:42295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:03.729619 7132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:03.730932 7132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:56:03.743034 6940 heartbeater.cc:503] Master 127.1.121.126:42295 was elected leader, sending a full tablet report...
W20260807 02:56:03.744436 7139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:03.744926 7142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:03.745281 7144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:03.746758 7132 server_base.cc:1034] running on GCE node
I20260807 02:56:03.747334 7132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:03.748245 7132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:03.749459 7132 hybrid_clock.cc:648] HybridClock initialized: now 1786071363749415 us; error 28 us; skew 500 ppm
I20260807 02:56:03.749806 7132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:03 dist-test-slave-3h5h krb5kdc[6601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:03.753093 7132 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:56:03.754673 7132 webserver.cc:460] Webserver started at http://127.1.121.67:42557/ using document root <none> and password file <none>
I20260807 02:56:03.755333 7132 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:03.755431 7132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:03.755697 7132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:03.758759 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "28332decfca443d8a1457e630bacc67c"
format_stamp: "Formatted at 2026-08-07 02:56:03 on dist-test-slave-3h5h"
server_key: "18c36e409ff376c7766ab1387f71d4ae"
server_key_iv: "ca01646b3568bd505db08e762d51d27e"
server_key_version: "encryptionkey@0"
I20260807 02:56:03.759579 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "28332decfca443d8a1457e630bacc67c"
format_stamp: "Formatted at 2026-08-07 02:56:03 on dist-test-slave-3h5h"
server_key: "18c36e409ff376c7766ab1387f71d4ae"
server_key_iv: "ca01646b3568bd505db08e762d51d27e"
server_key_version: "encryptionkey@0"
I20260807 02:56:03.764703 7132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 02:56:03.768226 7153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:03.769667 7132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:03.769842 7132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "28332decfca443d8a1457e630bacc67c"
format_stamp: "Formatted at 2026-08-07 02:56:03 on dist-test-slave-3h5h"
server_key: "18c36e409ff376c7766ab1387f71d4ae"
server_key_iv: "ca01646b3568bd505db08e762d51d27e"
server_key_version: "encryptionkey@0"
I20260807 02:56:03.770056 7132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:03.792523 7132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:03.797183 7132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:03.797494 7132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:03.798486 7132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:03.799894 7132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:03.800009 7132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:03.800173 7132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:03.800251 7132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:03.835006 7132 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:40039
I20260807 02:56:03.835099 7323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:40039 every 8 connection(s)
I20260807 02:56:03.836412 7132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:56:03.840806 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 7132
I20260807 02:56:03.841087 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:56:03.841528 1509 external_mini_cluster.cc:1428] Setting key 32e9446ab5d95ced5c409b12555bfe84
Aug 07 02:56:03 dist-test-slave-3h5h krb5kdc[6601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:03.857296 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:03.839006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:42415 (local address 127.1.121.126:42295)
0807 02:56:03.839468 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:03.839473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:03.840302 (+ 829us) server_negotiation.cc:407] Connection header received
0807 02:56:03.842458 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:03.842463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:03.842581 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:03.842717 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:03.844350 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.845295 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.847367 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.847761 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.847861 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:03.851209 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:03.851215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:03.851218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:03.851259 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:03.853923 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.854567 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.854579 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.854581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.854698 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.855082 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.855086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.855089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.855427 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:03.855601 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:03.856562 (+ 961us) server_negotiation.cc:299] Negotiation successful
0807 02:56:03.856781 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":252,"threads_started":1}
I20260807 02:56:03.858193 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:03.839414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42295 (local address 127.1.121.67:42415)
0807 02:56:03.840135 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:03.840177 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:03.842127 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:03.842979 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:03.842987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:03.843538 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:03.844174 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:03.844191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.845429 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.845432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:03.847236 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:03.847248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.847951 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.847955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:03.848149 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:03.848776 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:03.848800 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:03.851010 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:03.854045 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:03.854051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:03.854063 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:03.854467 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:03.854793 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:03.854796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:03.854798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:03.854969 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:03.855559 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:03.855565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:03.856054 (+ 489us) client_negotiation.cc:769] Sending connection context
0807 02:56:03.856399 (+ 345us) client_negotiation.cc:240] Negotiation successful
0807 02:56:03.856957 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":287,"threads_started":1}
I20260807 02:56:03.859766 7324 heartbeater.cc:347] Connected to a master server at 127.1.121.126:42295
I20260807 02:56:03.860078 7324 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:03.860816 7324 heartbeater.cc:511] Master 127.1.121.126:42295 requested a full tablet report, sending...
I20260807 02:56:03.862389 6670 ts_manager.cc:194] Registered new tserver with Master: 28332decfca443d8a1457e630bacc67c (127.1.121.67:40039)
I20260807 02:56:03.863231 6670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:42415
I20260807 02:56:03.869555 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:56:03 dist-test-slave-3h5h krb5kdc[6601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:56:03 dist-test-slave-3h5h krb5kdc[6601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:56:03.929977 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:03.912804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.1.121.65:38513)
0807 02:56:03.913587 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:03.913600 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:03.913641 (+ 41us) server_negotiation.cc:407] Connection header received
0807 02:56:03.913781 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:03.913797 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:03.914057 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:03.914275 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:03.915855 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.917074 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.918495 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.919128 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.919297 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:03.924044 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:03.924056 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:03.924074 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:03.924121 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:03.927408 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.928031 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.928038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.928040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.928211 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.928530 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.928533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.928536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.928972 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:03.929250 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:03.929446 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 02:56:03.929652 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":350,"threads_started":1}
Aug 07 02:56:03 dist-test-slave-3h5h krb5kdc[6601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:03.947870 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:03.934623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.1.121.126:42295)
0807 02:56:03.934850 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:03.934856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:03.935033 (+ 177us) server_negotiation.cc:407] Connection header received
0807 02:56:03.935237 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:03.935241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:03.935319 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:03.935398 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:03.936713 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.937496 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.938702 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.939082 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.939189 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:03.943046 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:03.943051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:03.943054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:03.943084 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:03.945583 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.946177 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.946181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.946184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.946305 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.946617 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.946620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.946622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.946920 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:03.947100 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:03.947354 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 02:56:03.947516 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260807 02:56:03.948549 6670 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43440
I20260807 02:56:03.949651 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 6757
I20260807 02:56:03.963335 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:03.950583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.1.121.65:38513)
0807 02:56:03.950803 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:03.950809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:03.950997 (+ 188us) server_negotiation.cc:407] Connection header received
0807 02:56:03.951208 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:03.951212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:03.951303 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:03.951411 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:03.952672 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.953434 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.954732 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:03.955157 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:03.955301 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:03.957682 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:03.957687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:03.957690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:03.957724 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:03.960646 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.961343 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.961350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.961354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.961489 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:03.961887 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:03.961892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:03.961896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:03.962178 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:03.962433 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:03.962744 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 02:56:03.962978 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 02:56:03.964069 6933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:04.049201 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 6757
I20260807 02:56:04.075950 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 6946
Aug 07 02:56:04 dist-test-slave-3h5h krb5kdc[6601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:56:04.092718 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:04.077029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.1.121.66:33427)
0807 02:56:04.077679 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:04.077686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:04.077715 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:56:04.077788 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:04.077794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:04.078013 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:04.078158 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:04.079362 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:04.080390 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:04.081712 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:04.082265 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:04.082368 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:04.087007 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:04.087020 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:04.087037 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:04.087072 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:04.089958 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:04.090743 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:04.090750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:04.090753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:04.090876 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:04.091265 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:04.091268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:04.091270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:04.091619 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:04.091909 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:04.092182 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 02:56:04.092386 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":335,"threads_started":1}
I20260807 02:56:04.094120 7114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:04.179857 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 6946
I20260807 02:56:04.204185 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 7132
Aug 07 02:56:04 dist-test-slave-3h5h krb5kdc[6601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:56:04.221123 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:04.205323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.1.121.67:40039)
0807 02:56:04.205967 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:04.205975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:04.206004 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:56:04.206065 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:04.206071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:04.206288 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:04.206414 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:04.207712 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:04.208735 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:04.210019 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:04.210575 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:04.210683 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:04.215409 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:04.215422 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:04.215439 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:04.215479 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:04.218482 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:04.219108 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:04.219115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:04.219117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:04.219259 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:04.219577 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:04.219580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:04.219582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:04.219926 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:04.220186 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:04.220407 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 02:56:04.220650 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":331,"threads_started":1}
I20260807 02:56:04.222556 7317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:04.306967 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 7132
I20260807 02:56:04.332412 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 6622
I20260807 02:56:04.333889 6717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:04.444695 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 6622
2026-08-07T02:56:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4456 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:56:04 dist-test-slave-3h5h krb5kdc[7351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:56:04 dist-test-slave-3h5h krb5kdc[7351](info): set up 2 sockets
Aug 07 02:56:04 dist-test-slave-3h5h krb5kdc[7351](info): commencing operation
krb5kdc: starting...
W20260807 02:56:05.797637 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:56:05 dist-test-slave-3h5h krb5kdc[7351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:56:06.048542 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45251
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:06.397764 7368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:06.398123 7368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:06.398293 7368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:06.405609 7368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:06.405763 7368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:06.405876 7368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:06.405973 7368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:06.406067 7368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:06.416162 7368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:45251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:06.417661 7368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:06.418993 7368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:06.431140 7377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:06.431196 7382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:06.431738 7375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:06.431919 7368 server_base.cc:1034] running on GCE node
I20260807 02:56:06.432797 7368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:06.434254 7368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:06.435465 7368 hybrid_clock.cc:648] HybridClock initialized: now 1786071366435419 us; error 36 us; skew 500 ppm
I20260807 02:56:06.435850 7368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:06 dist-test-slave-3h5h krb5kdc[7351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:06.439101 7368 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:56:06.440239 7368 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:06.440330 7368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:06.440598 7368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:06.443647 7368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "236561cedb814a309bb12091eec30a34"
format_stamp: "Formatted at 2026-08-07 02:56:06 on dist-test-slave-3h5h"
server_key: "5a68d9d13ed74d9a182b4b4dd45ef734"
server_key_iv: "07ad189ea36f64046c2167b7e0ecf685"
server_key_version: "encryptionkey@0"
I20260807 02:56:06.444386 7368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "236561cedb814a309bb12091eec30a34"
format_stamp: "Formatted at 2026-08-07 02:56:06 on dist-test-slave-3h5h"
server_key: "5a68d9d13ed74d9a182b4b4dd45ef734"
server_key_iv: "07ad189ea36f64046c2167b7e0ecf685"
server_key_version: "encryptionkey@0"
I20260807 02:56:06.449472 7368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 02:56:06.452960 7388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:06.454274 7368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:56:06.454463 7368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "236561cedb814a309bb12091eec30a34"
format_stamp: "Formatted at 2026-08-07 02:56:06 on dist-test-slave-3h5h"
server_key: "5a68d9d13ed74d9a182b4b4dd45ef734"
server_key_iv: "07ad189ea36f64046c2167b7e0ecf685"
server_key_version: "encryptionkey@0"
I20260807 02:56:06.454694 7368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:06.468039 7368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:06.471985 7368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:06.472273 7368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:06.491477 7368 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:45251
I20260807 02:56:06.491495 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:45251 every 8 connection(s)
I20260807 02:56:06.492424 7368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:06.493625 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 7368
I20260807 02:56:06.493912 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:06.494745 1509 external_mini_cluster.cc:1428] Setting key 7042f3fb14fd67b032016167fe74dd1e
I20260807 02:56:06.497270 7459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 02:56:06 dist-test-slave-3h5h krb5kdc[7351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:06.510974 7459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34: Bootstrap starting.
I20260807 02:56:06.515447 7459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:06.516711 7459 log.cc:824] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:06.520212 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:06.496958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.1.121.126:45251)
0807 02:56:06.497783 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:06.497794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:06.497835 (+ 41us) server_negotiation.cc:407] Connection header received
0807 02:56:06.499537 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:06.499559 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:06.499940 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:06.500256 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:06.502169 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:06.503473 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:06.505311 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:06.506010 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:06.506154 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:06.510929 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:06.510943 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:06.510971 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:06.511028 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:06.515117 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:06.515996 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:06.516004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:06.516012 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:06.516225 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:06.516668 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:06.516673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:06.516676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:06.517552 (+ 876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:06.517795 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:06.518374 (+ 579us) server_negotiation.cc:299] Negotiation successful
0807 02:56:06.518894 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":400,"threads_started":1}
I20260807 02:56:06.520442 7459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34: No bootstrap required, opened a new log
I20260807 02:56:06.528573 7459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236561cedb814a309bb12091eec30a34" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45251 } }
I20260807 02:56:06.528977 7459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:06.529062 7459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 236561cedb814a309bb12091eec30a34, State: Initialized, Role: FOLLOWER
I20260807 02:56:06.529549 7459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236561cedb814a309bb12091eec30a34" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45251 } }
I20260807 02:56:06.529760 7459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:06.529858 7459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:06.529979 7459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:06.532091 7459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236561cedb814a309bb12091eec30a34" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45251 } }
I20260807 02:56:06.532739 7459 leader_election.cc:302] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 236561cedb814a309bb12091eec30a34; no voters:
I20260807 02:56:06.533360 7459 leader_election.cc:288] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:06.533499 7467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:06.534389 7467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [term 1 LEADER]: Becoming Leader. State: Replica: 236561cedb814a309bb12091eec30a34, State: Running, Role: LEADER
I20260807 02:56:06.534937 7467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236561cedb814a309bb12091eec30a34" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45251 } }
I20260807 02:56:06.535558 7459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:06.538335 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "236561cedb814a309bb12091eec30a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236561cedb814a309bb12091eec30a34" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45251 } } }
I20260807 02:56:06.538347 7470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 236561cedb814a309bb12091eec30a34. Latest consensus state: current_term: 1 leader_uuid: "236561cedb814a309bb12091eec30a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "236561cedb814a309bb12091eec30a34" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45251 } } }
I20260807 02:56:06.538738 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:06.538810 7470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:06.540903 7482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:06.546713 7482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:06.555609 7482 catalog_manager.cc:1347] Generated new cluster ID: 4db9820f318542c6acf77072c0518a0e
I20260807 02:56:06.555732 7482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:06.584635 7482 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:06.586714 7482 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:06.603313 7482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34: Generated new TSK 0
I20260807 02:56:06.604435 7482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:56:06.665256 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45251
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:07.017123 7500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:07.017405 7500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:07.017601 7500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:07.024945 7500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:07.025053 7500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:07.025264 7500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:56:07.036072 7500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:07.037643 7500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:07.039528 7500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:07.053217 7513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:07.053313 7510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:07.053287 7509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:07.054080 7500 server_base.cc:1034] running on GCE node
I20260807 02:56:07.054710 7500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:07.055848 7500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:07.057085 7500 hybrid_clock.cc:648] HybridClock initialized: now 1786071367057019 us; error 34 us; skew 500 ppm
I20260807 02:56:07.057430 7500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:07 dist-test-slave-3h5h krb5kdc[7351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071367, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:07.064070 7500 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:56:07.065754 7500 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:07.065900 7500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:07.066311 7500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:07.069432 7500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "723ebbff80d44899b9dcb84b19eb7e9a"
format_stamp: "Formatted at 2026-08-07 02:56:07 on dist-test-slave-3h5h"
server_key: "71d86fdf0bccf5a5faa94d619fba2ae9"
server_key_iv: "0daabd85e0685a8d7875e55c20834a0f"
server_key_version: "encryptionkey@0"
I20260807 02:56:07.070290 7500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "723ebbff80d44899b9dcb84b19eb7e9a"
format_stamp: "Formatted at 2026-08-07 02:56:07 on dist-test-slave-3h5h"
server_key: "71d86fdf0bccf5a5faa94d619fba2ae9"
server_key_iv: "0daabd85e0685a8d7875e55c20834a0f"
server_key_version: "encryptionkey@0"
I20260807 02:56:07.075737 7500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 02:56:07.079387 7527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:07.080754 7500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:56:07.080962 7500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "723ebbff80d44899b9dcb84b19eb7e9a"
format_stamp: "Formatted at 2026-08-07 02:56:07 on dist-test-slave-3h5h"
server_key: "71d86fdf0bccf5a5faa94d619fba2ae9"
server_key_iv: "0daabd85e0685a8d7875e55c20834a0f"
server_key_version: "encryptionkey@0"
I20260807 02:56:07.081197 7500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:07.102864 7500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:07.107112 7500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:07.107404 7500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:07.108397 7500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:07.109923 7500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:07.110057 7500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:07.110186 7500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:07.110275 7500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:07.146245 7500 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:44767
I20260807 02:56:07.146334 7677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:44767 every 8 connection(s)
I20260807 02:56:07.147203 7500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:56:07.150903 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 7500
I20260807 02:56:07.151152 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:56:07.152047 1509 external_mini_cluster.cc:1428] Setting key 5bf245f521e6df8fd083674bb59000c3
Aug 07 02:56:07 dist-test-slave-3h5h krb5kdc[7351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071367, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:07.168720 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:07.149959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:37205 (local address 127.1.121.126:45251)
0807 02:56:07.150442 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:07.150448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:07.151726 (+ 1278us) server_negotiation.cc:407] Connection header received
0807 02:56:07.153960 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:07.153965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:07.154057 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:07.154180 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:07.155694 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:07.156405 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:07.158097 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:07.158585 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:07.158720 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:07.162430 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:07.162435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:07.162439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:07.162471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:07.165186 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:07.165984 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:07.165990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:07.165995 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:07.166161 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:07.166742 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:07.166746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:07.166748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:07.167090 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:07.167214 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:07.168264 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0807 02:56:07.168453 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260807 02:56:07.170333 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:07.150569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45251 (local address 127.1.121.65:37205)
0807 02:56:07.151541 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:07.151593 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:07.153670 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:07.154421 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:07.154429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:07.154952 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:07.155521 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:07.155540 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:07.156584 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:07.156590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:07.157926 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:07.157940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:07.158823 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:07.158833 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:07.159066 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:07.159796 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:07.159818 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:07.162205 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:07.165373 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:07.165382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:07.165399 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:07.165850 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:07.166327 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:07.166335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:07.166342 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:07.166598 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:07.167253 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:07.167266 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:07.167794 (+ 528us) client_negotiation.cc:769] Sending connection context
0807 02:56:07.168116 (+ 322us) client_negotiation.cc:240] Negotiation successful
0807 02:56:07.168733 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":337,"threads_started":1}
I20260807 02:56:07.172194 7678 heartbeater.cc:347] Connected to a master server at 127.1.121.126:45251
I20260807 02:56:07.172598 7678 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:07.173518 7678 heartbeater.cc:511] Master 127.1.121.126:45251 requested a full tablet report, sending...
I20260807 02:56:07.175807 7410 ts_manager.cc:194] Registered new tserver with Master: 723ebbff80d44899b9dcb84b19eb7e9a (127.1.121.65:44767)
I20260807 02:56:07.177325 7410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:37205
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:56:07.214656 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45251
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:07.570729 7685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:07.571022 7685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:07.571197 7685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:07.578529 7685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:07.578697 7685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:07.578888 7685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:56:07.589052 7685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:07.590490 7685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:07.591850 7685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:07.604697 7692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:07.605262 7693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:07.605931 7699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:07.606223 7685 server_base.cc:1034] running on GCE node
I20260807 02:56:07.606866 7685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:07.607832 7685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:07.609062 7685 hybrid_clock.cc:648] HybridClock initialized: now 1786071367609003 us; error 67 us; skew 500 ppm
I20260807 02:56:07.609382 7685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:07 dist-test-slave-3h5h krb5kdc[7351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071367, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:07.612465 7685 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:56:07.613600 7685 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:07.613710 7685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:07.613982 7685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:07.617977 7685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "bb213d5c80a14edd8af2c8b8200a54f3"
format_stamp: "Formatted at 2026-08-07 02:56:07 on dist-test-slave-3h5h"
server_key: "13f2a1c643a3c8ab61fe10ba0ad93ee6"
server_key_iv: "f0c9a490009a0be32d24bf29ae7e2a03"
server_key_version: "encryptionkey@0"
I20260807 02:56:07.618757 7685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "bb213d5c80a14edd8af2c8b8200a54f3"
format_stamp: "Formatted at 2026-08-07 02:56:07 on dist-test-slave-3h5h"
server_key: "13f2a1c643a3c8ab61fe10ba0ad93ee6"
server_key_iv: "f0c9a490009a0be32d24bf29ae7e2a03"
server_key_version: "encryptionkey@0"
I20260807 02:56:07.623996 7685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 02:56:07.627588 7709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:07.629005 7685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 02:56:07.629143 7685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "bb213d5c80a14edd8af2c8b8200a54f3"
format_stamp: "Formatted at 2026-08-07 02:56:07 on dist-test-slave-3h5h"
server_key: "13f2a1c643a3c8ab61fe10ba0ad93ee6"
server_key_iv: "f0c9a490009a0be32d24bf29ae7e2a03"
server_key_version: "encryptionkey@0"
I20260807 02:56:07.629323 7685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:07.646548 7685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:07.650857 7685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:07.651094 7685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:07.652005 7685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:07.653453 7685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:07.653556 7685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:07.653632 7685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:07.653712 7685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:07.688627 7685 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:37081
I20260807 02:56:07.688726 7862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:37081 every 8 connection(s)
I20260807 02:56:07.689538 7685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:56:07.694509 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 7685
I20260807 02:56:07.694723 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:56:07.695286 1509 external_mini_cluster.cc:1428] Setting key 39d88bec6989e2814bd43a9020f314cc
Aug 07 02:56:07 dist-test-slave-3h5h krb5kdc[7351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071367, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:07.712033 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:07.692123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:58103 (local address 127.1.121.126:45251)
0807 02:56:07.692569 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:07.692576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:07.693582 (+ 1006us) server_negotiation.cc:407] Connection header received
0807 02:56:07.696085 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:07.696092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:07.696184 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:07.696292 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:07.698205 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:07.698977 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:07.701463 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:07.701897 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:07.701993 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:07.706294 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:07.706299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:07.706302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:07.706336 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:07.708974 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:07.709714 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:07.709723 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:07.709726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:07.709851 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:07.710306 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:07.710310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:07.710312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:07.710567 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:07.710735 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:07.711430 (+ 695us) server_negotiation.cc:299] Negotiation successful
0807 02:56:07.711648 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":218,"threads_started":1}
I20260807 02:56:07.713079 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:07.692681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45251 (local address 127.1.121.66:58103)
0807 02:56:07.693426 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:07.693464 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:07.695751 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:07.696553 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:07.696561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:07.697227 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:07.697997 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:07.698033 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:07.699151 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:07.699160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:07.701319 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:07.701339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:07.702129 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:07.702140 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:07.702471 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:07.703251 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:07.703286 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:07.706034 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:07.709113 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:07.709122 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:07.709138 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:07.709573 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:07.709968 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:07.709974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:07.709978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:07.710176 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:07.710709 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:07.710717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:07.711053 (+ 336us) client_negotiation.cc:769] Sending connection context
0807 02:56:07.711323 (+ 270us) client_negotiation.cc:240] Negotiation successful
0807 02:56:07.711857 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":337,"threads_started":1}
I20260807 02:56:07.714636 7863 heartbeater.cc:347] Connected to a master server at 127.1.121.126:45251
I20260807 02:56:07.714984 7863 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:07.715848 7863 heartbeater.cc:511] Master 127.1.121.126:45251 requested a full tablet report, sending...
I20260807 02:56:07.717622 7410 ts_manager.cc:194] Registered new tserver with Master: bb213d5c80a14edd8af2c8b8200a54f3 (127.1.121.66:37081)
I20260807 02:56:07.719077 7410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:58103
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:56:07.757488 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45251
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:08.106498 7873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:08.106807 7873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:08.107012 7873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:08.114707 7873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:08.114864 7873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:08.115082 7873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:56:08.125658 7873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:08.127148 7873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:08.128566 7873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:08.142715 7880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:08.142740 7881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:08.142818 7886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:08.143620 7873 server_base.cc:1034] running on GCE node
I20260807 02:56:08.144306 7873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:08.145367 7873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:08.146572 7873 hybrid_clock.cc:648] HybridClock initialized: now 1786071368146511 us; error 55 us; skew 500 ppm
I20260807 02:56:08.146904 7873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:08 dist-test-slave-3h5h krb5kdc[7351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071368, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:08.150295 7873 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:56:08.151783 7873 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:08.151877 7873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:08.152240 7873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:08.155277 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0"
format_stamp: "Formatted at 2026-08-07 02:56:08 on dist-test-slave-3h5h"
server_key: "586a021f5a2241fb175d78a431a612e0"
server_key_iv: "e918994c7548b9fb3b7aa051b07be315"
server_key_version: "encryptionkey@0"
I20260807 02:56:08.156112 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0"
format_stamp: "Formatted at 2026-08-07 02:56:08 on dist-test-slave-3h5h"
server_key: "586a021f5a2241fb175d78a431a612e0"
server_key_iv: "e918994c7548b9fb3b7aa051b07be315"
server_key_version: "encryptionkey@0"
I20260807 02:56:08.161686 7873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 02:56:08.165123 7904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:08.166509 7873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:56:08.166687 7873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0"
format_stamp: "Formatted at 2026-08-07 02:56:08 on dist-test-slave-3h5h"
server_key: "586a021f5a2241fb175d78a431a612e0"
server_key_iv: "e918994c7548b9fb3b7aa051b07be315"
server_key_version: "encryptionkey@0"
I20260807 02:56:08.166925 7873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:08.180222 7678 heartbeater.cc:503] Master 127.1.121.126:45251 was elected leader, sending a full tablet report...
I20260807 02:56:08.197949 7873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:08.202397 7873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:08.202715 7873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:08.203723 7873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:08.205179 7873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:08.205312 7873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:08.205437 7873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:08.205526 7873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:08.238600 7873 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:45415
I20260807 02:56:08.238685 8074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:45415 every 8 connection(s)
I20260807 02:56:08.239508 7873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:56:08.242767 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 7873
I20260807 02:56:08.243000 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:56:08.243589 1509 external_mini_cluster.cc:1428] Setting key 7240283570086bd13d77528e1b8c38ca
Aug 07 02:56:08 dist-test-slave-3h5h krb5kdc[7351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071368, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:08.260311 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.242054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:45487 (local address 127.1.121.126:45251)
0807 02:56:08.242650 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.242654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.243580 (+ 926us) server_negotiation.cc:407] Connection header received
0807 02:56:08.245981 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.245984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.246064 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.246159 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:08.247806 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.248558 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.250321 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.250721 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.250814 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.254093 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:08.254098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:08.254101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:08.254133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:08.256951 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.257818 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.257822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.257824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.257990 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.258542 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.258546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.258549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.258910 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:08.259057 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.259859 (+ 802us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.260027 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":384,"threads_started":1}
I20260807 02:56:08.261312 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.242582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45251 (local address 127.1.121.67:45487)
0807 02:56:08.243418 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:08.243455 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:08.245690 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:08.246434 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:08.246446 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:08.247092 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:08.247644 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.247662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.248704 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.248707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.250188 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.250200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.250902 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.250907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.251082 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.251698 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:08.251719 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:08.253888 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:08.257109 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:08.257118 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:08.257134 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:08.257659 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:08.258114 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:08.258120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:08.258123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:08.258381 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:08.259053 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:08.259063 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:08.259509 (+ 446us) client_negotiation.cc:769] Sending connection context
0807 02:56:08.259736 (+ 227us) client_negotiation.cc:240] Negotiation successful
0807 02:56:08.260127 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":371,"threads_started":1}
I20260807 02:56:08.262640 8075 heartbeater.cc:347] Connected to a master server at 127.1.121.126:45251
I20260807 02:56:08.262956 8075 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:08.263610 8075 heartbeater.cc:511] Master 127.1.121.126:45251 requested a full tablet report, sending...
I20260807 02:56:08.265008 7410 ts_manager.cc:194] Registered new tserver with Master: eb750b23c51c44e6a5c9e9d6f70e59b0 (127.1.121.67:45415)
I20260807 02:56:08.265872 7410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:45487
I20260807 02:56:08.271631 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:56:08.288466 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.276218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.1.121.126:45251)
0807 02:56:08.276385 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.276389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.277056 (+ 667us) server_negotiation.cc:407] Connection header received
0807 02:56:08.277207 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.277211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.277286 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.277365 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:08.278763 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.279462 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.280641 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.281102 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.281201 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.283486 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:08.283491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:08.283494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:08.283520 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:08.286151 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.286753 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.286757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.286759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.286874 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.287222 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.287225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.287227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.287612 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:08.287808 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.288074 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.288234 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 02:56:08.292645 7410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:56:08.299790 7410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:56:08.335182 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.312764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.1.121.66:37081)
0807 02:56:08.326113 (+ 13349us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.326121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.326164 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:56:08.326255 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.326262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.326488 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.327227 (+ 739us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:08.327687 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.329328 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.332190 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.333858 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.334008 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.334225 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.334629 (+ 404us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.334817 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13182,"thread_start_us":13245,"threads_started":1}
I20260807 02:56:08.335350 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.312482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:37081 (local address 127.0.0.1:41772)
0807 02:56:08.319628 (+ 7146us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:08.319679 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:08.319890 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:08.326714 (+ 6824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:08.326718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:08.326750 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:08.327403 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.327419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.329490 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.329495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.332018 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.332033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.334095 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.334104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.334340 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.334405 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 02:56:08.334558 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 02:56:08.334805 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6795,"mutex_wait_us":138,"thread_start_us":375,"threads_started":1}
I20260807 02:56:08.336156 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.313511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.1.121.67:45415)
0807 02:56:08.325839 (+ 12328us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.325849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.325886 (+ 37us) server_negotiation.cc:407] Connection header received
0807 02:56:08.325991 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.326003 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.326315 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.326536 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:08.327554 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.329329 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.332604 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.334213 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.334362 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.334613 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.335452 (+ 839us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.335656 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12097,"thread_start_us":11920,"threads_started":1}
I20260807 02:56:08.336820 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.313120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:45415 (local address 127.0.0.1:34152)
0807 02:56:08.324955 (+ 11835us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:08.324981 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:08.325143 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:08.326563 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:08.326581 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:08.326629 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:08.327275 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.327301 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.329528 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.329532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.332427 (+ 2895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.332439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.334772 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.334778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.335001 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.335038 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 02:56:08.335151 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 02:56:08.335318 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11626,"mutex_wait_us":203,"thread_start_us":11471,"threads_started":1}
I20260807 02:56:08.337190 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.315444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:44767 (local address 127.0.0.1:40512)
0807 02:56:08.325401 (+ 9957us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:08.325423 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:08.325536 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:08.326923 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:08.326928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:08.326953 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:08.327702 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.327717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.329631 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.329635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.332336 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.332351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.334392 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.334401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.334583 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.334616 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 02:56:08.334729 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 02:56:08.334852 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9800,"mutex_wait_us":155,"thread_start_us":9614,"threads_started":1}
I20260807 02:56:08.337746 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.314605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.1.121.65:44767)
0807 02:56:08.326112 (+ 11507us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.326121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.326153 (+ 32us) server_negotiation.cc:407] Connection header received
0807 02:56:08.326254 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.326262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.326481 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.326682 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:08.327913 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.329480 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.332529 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.334047 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.334171 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.334382 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.335014 (+ 632us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.337501 (+ 2487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11322,"thread_start_us":11146,"threads_started":1}
I20260807 02:56:08.340190 7774 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec7d33b5c9149ef840f225ba409ef60 (DEFAULT_TABLE table=test-table [id=e1c0b436d2004e56990040e7c258fd6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:08.340587 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec7d33b5c9149ef840f225ba409ef60 (DEFAULT_TABLE table=test-table [id=e1c0b436d2004e56990040e7c258fd6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:08.340631 7982 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec7d33b5c9149ef840f225ba409ef60 (DEFAULT_TABLE table=test-table [id=e1c0b436d2004e56990040e7c258fd6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:08.341873 7774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7d33b5c9149ef840f225ba409ef60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:08.342060 7982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7d33b5c9149ef840f225ba409ef60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:08.342062 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7d33b5c9149ef840f225ba409ef60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:08.353031 8102 tablet_bootstrap.cc:492] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: Bootstrap starting.
I20260807 02:56:08.357245 8102 tablet_bootstrap.cc:654] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:08.358970 8102 log.cc:824] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:08.359464 8104 tablet_bootstrap.cc:492] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: Bootstrap starting.
I20260807 02:56:08.360169 8105 tablet_bootstrap.cc:492] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: Bootstrap starting.
I20260807 02:56:08.362469 8102 tablet_bootstrap.cc:492] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: No bootstrap required, opened a new log
I20260807 02:56:08.362778 8102 ts_tablet_manager.cc:1397] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260807 02:56:08.364506 8105 tablet_bootstrap.cc:654] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:08.364984 8104 tablet_bootstrap.cc:654] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:08.366127 8105 log.cc:824] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:08.366403 8104 log.cc:824] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:08.369812 8105 tablet_bootstrap.cc:492] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: No bootstrap required, opened a new log
I20260807 02:56:08.370163 8104 tablet_bootstrap.cc:492] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: No bootstrap required, opened a new log
I20260807 02:56:08.370169 8105 ts_tablet_manager.cc:1397] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260807 02:56:08.370471 8104 ts_tablet_manager.cc:1397] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260807 02:56:08.373780 8102 raft_consensus.cc:348] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } }
I20260807 02:56:08.374351 8102 raft_consensus.cc:374] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:08.374454 8102 raft_consensus.cc:729] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723ebbff80d44899b9dcb84b19eb7e9a, State: Initialized, Role: FOLLOWER
I20260807 02:56:08.375177 8102 consensus_queue.cc:260] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } }
I20260807 02:56:08.377429 8102 ts_tablet_manager.cc:1428] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260807 02:56:08.382699 8104 raft_consensus.cc:348] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } }
I20260807 02:56:08.382766 8105 raft_consensus.cc:348] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } }
I20260807 02:56:08.383204 8105 raft_consensus.cc:374] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:08.383204 8104 raft_consensus.cc:374] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:08.383311 8105 raft_consensus.cc:729] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb750b23c51c44e6a5c9e9d6f70e59b0, State: Initialized, Role: FOLLOWER
I20260807 02:56:08.383311 8104 raft_consensus.cc:729] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb213d5c80a14edd8af2c8b8200a54f3, State: Initialized, Role: FOLLOWER
I20260807 02:56:08.384013 8104 consensus_queue.cc:260] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } }
I20260807 02:56:08.383996 8105 consensus_queue.cc:260] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } }
I20260807 02:56:08.385203 8113 raft_consensus.cc:482] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:08.385639 8075 heartbeater.cc:503] Master 127.1.121.126:45251 was elected leader, sending a full tablet report...
I20260807 02:56:08.385522 8113 raft_consensus.cc:504] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } }
I20260807 02:56:08.387688 7863 heartbeater.cc:503] Master 127.1.121.126:45251 was elected leader, sending a full tablet report...
I20260807 02:56:08.388500 8104 ts_tablet_manager.cc:1428] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260807 02:56:08.390267 8105 ts_tablet_manager.cc:1428] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: Time spent starting tablet: real 0.020s user 0.011s sys 0.004s
I20260807 02:56:08.390846 8113 leader_election.cc:288] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb213d5c80a14edd8af2c8b8200a54f3 (127.1.121.66:37081), eb750b23c51c44e6a5c9e9d6f70e59b0 (127.1.121.67:45415)
I20260807 02:56:08.399277 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.390293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:54295 (local address 127.1.121.66:37081)
0807 02:56:08.391380 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.391387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.391411 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:56:08.391498 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.391504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.391585 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.391697 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:08.392994 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.394244 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.397057 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.398195 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.398401 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.398484 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.398843 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.398979 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924}
I20260807 02:56:08.399405 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.389667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:37081 (local address 127.1.121.65:54295)
0807 02:56:08.390336 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:08.390360 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:08.390516 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:08.392120 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:08.392128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:08.392164 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:08.392727 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.392743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.394398 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.394404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.396793 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.396810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.398419 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.398426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.398629 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.398660 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 02:56:08.398780 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 02:56:08.398924 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":23,"thread_start_us":691,"threads_started":1}
I20260807 02:56:08.399422 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.391377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:45695 (local address 127.1.121.67:45415)
0807 02:56:08.391639 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.391647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.392288 (+ 641us) server_negotiation.cc:407] Connection header received
0807 02:56:08.392507 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.392517 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.392607 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.392721 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:08.393722 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.394826 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.397672 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.398425 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.398604 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.398695 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.399031 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.399157 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:56:08.399577 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.391111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:45415 (local address 127.1.121.65:45695)
0807 02:56:08.392149 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:08.392173 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:08.392336 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:08.392949 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:08.392955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:08.392988 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:08.393428 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.393458 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.394984 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.394990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.397497 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:08.397515 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.398672 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.398679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:08.398827 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.398890 (+ 63us) client_negotiation.cc:769] Sending connection context
0807 02:56:08.399005 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 02:56:08.399142 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":57,"thread_start_us":737,"threads_started":1}
W20260807 02:56:08.399840 7679 tablet.cc:2369] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:56:08.400665 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7d33b5c9149ef840f225ba409ef60" candidate_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" is_pre_election: true
I20260807 02:56:08.400660 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7d33b5c9149ef840f225ba409ef60" candidate_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" is_pre_election: true
I20260807 02:56:08.401309 8016 raft_consensus.cc:2454] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723ebbff80d44899b9dcb84b19eb7e9a in term 0.
I20260807 02:56:08.401309 7800 raft_consensus.cc:2454] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723ebbff80d44899b9dcb84b19eb7e9a in term 0.
I20260807 02:56:08.402050 7529 leader_election.cc:302] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 723ebbff80d44899b9dcb84b19eb7e9a, eb750b23c51c44e6a5c9e9d6f70e59b0; no voters:
I20260807 02:56:08.402500 8113 raft_consensus.cc:2781] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:56:08.402669 8113 raft_consensus.cc:482] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:56:08.402788 8113 raft_consensus.cc:3037] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:08.405715 8113 raft_consensus.cc:504] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } }
I20260807 02:56:08.406402 8113 leader_election.cc:288] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [CANDIDATE]: Term 1 election: Requested vote from peers bb213d5c80a14edd8af2c8b8200a54f3 (127.1.121.66:37081), eb750b23c51c44e6a5c9e9d6f70e59b0 (127.1.121.67:45415)
I20260807 02:56:08.406919 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7d33b5c9149ef840f225ba409ef60" candidate_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb213d5c80a14edd8af2c8b8200a54f3"
I20260807 02:56:08.407018 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7d33b5c9149ef840f225ba409ef60" candidate_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0"
I20260807 02:56:08.407155 7800 raft_consensus.cc:3037] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:08.407276 8016 raft_consensus.cc:3037] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:08.409672 8016 raft_consensus.cc:2454] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723ebbff80d44899b9dcb84b19eb7e9a in term 1.
I20260807 02:56:08.409788 7800 raft_consensus.cc:2454] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723ebbff80d44899b9dcb84b19eb7e9a in term 1.
I20260807 02:56:08.410130 7529 leader_election.cc:302] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 723ebbff80d44899b9dcb84b19eb7e9a, eb750b23c51c44e6a5c9e9d6f70e59b0; no voters:
I20260807 02:56:08.410519 8113 raft_consensus.cc:2781] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:08.410707 8113 raft_consensus.cc:686] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 1 LEADER]: Becoming Leader. State: Replica: 723ebbff80d44899b9dcb84b19eb7e9a, State: Running, Role: LEADER
I20260807 02:56:08.411322 8113 consensus_queue.cc:237] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } }
I20260807 02:56:08.416561 7410 catalog_manager.cc:5495] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 723ebbff80d44899b9dcb84b19eb7e9a (127.1.121.65). New cstate: current_term: 1 leader_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723ebbff80d44899b9dcb84b19eb7e9a" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 44767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 } health_report { overall_health: UNKNOWN } } }
I20260807 02:56:08.440526 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.433581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.1.121.65:44767)
0807 02:56:08.433871 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.433879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.434044 (+ 165us) server_negotiation.cc:407] Connection header received
0807 02:56:08.434348 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.434353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.434472 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.434664 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:56:08.435238 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.436162 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.437850 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.438356 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.438536 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.439169 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:56:08.439742 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:56:08.439950 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.440052 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.440155 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 02:56:08.441545 7867 tablet.cc:2369] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:56:08.449949 7800 raft_consensus.cc:1261] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [term 1 FOLLOWER]: Refusing update from remote peer 723ebbff80d44899b9dcb84b19eb7e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:08.449952 8016 raft_consensus.cc:1261] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [term 1 FOLLOWER]: Refusing update from remote peer 723ebbff80d44899b9dcb84b19eb7e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:08.451159 8112 consensus_queue.cc:1035] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb750b23c51c44e6a5c9e9d6f70e59b0" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 45415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:08.451524 8113 consensus_queue.cc:1035] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb213d5c80a14edd8af2c8b8200a54f3" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 37081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:08.463864 8121 mvcc.cc:205] Tried to move back new op lower bound from 7315748325159292928 to 7315748325016965120. Current Snapshot: MvccSnapshot[applied={T|T < 7315748325159292928}]
I20260807 02:56:08.481549 7409 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:08.481892 7409 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:08.486296 7409 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 236561cedb814a309bb12091eec30a34: Sending DeleteTablet for 3 replicas of tablet 0ec7d33b5c9149ef840f225ba409ef60
I20260807 02:56:08.487532 7774 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec7d33b5c9149ef840f225ba409ef60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:08 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:41772
I20260807 02:56:08.487727 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec7d33b5c9149ef840f225ba409ef60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:08 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:40512
I20260807 02:56:08.487854 7982 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec7d33b5c9149ef840f225ba409ef60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:08 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:34152
I20260807 02:56:08.490550 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 7500
I20260807 02:56:08.498646 8138 tablet_replica.cc:330] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: stopping tablet replica
I20260807 02:56:08.499147 8138 raft_consensus.cc:2229] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:08.499764 8138 raft_consensus.cc:2258] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:08.505272 8136 tablet_replica.cc:330] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: stopping tablet replica
I20260807 02:56:08.505735 8136 raft_consensus.cc:2229] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:08.506296 8136 raft_consensus.cc:2258] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:08.507164 8137 tablet_replica.cc:330] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: stopping tablet replica
I20260807 02:56:08.507593 8137 raft_consensus.cc:2229] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:56:08.508371 8137 raft_consensus.cc:2258] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:08.510427 8136 ts_tablet_manager.cc:1905] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:08.510399 8138 ts_tablet_manager.cc:1905] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:08.517689 8136 ts_tablet_manager.cc:1918] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:08.517886 8136 log.cc:1196] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/0ec7d33b5c9149ef840f225ba409ef60
I20260807 02:56:08.518523 8136 ts_tablet_manager.cc:1939] T 0ec7d33b5c9149ef840f225ba409ef60 P bb213d5c80a14edd8af2c8b8200a54f3: Deleting consensus metadata
Aug 07 02:56:08 dist-test-slave-3h5h krb5kdc[7351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:56:08.520565 8137 ts_tablet_manager.cc:1905] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:08.520517 7392 catalog_manager.cc:4841] TS bb213d5c80a14edd8af2c8b8200a54f3 (127.1.121.66:37081): tablet 0ec7d33b5c9149ef840f225ba409ef60 (table test-table [id=e1c0b436d2004e56990040e7c258fd6f]) successfully deleted
I20260807 02:56:08.526623 8138 ts_tablet_manager.cc:1918] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:08.526827 8138 log.cc:1196] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/0ec7d33b5c9149ef840f225ba409ef60
I20260807 02:56:08.527403 8138 ts_tablet_manager.cc:1939] T 0ec7d33b5c9149ef840f225ba409ef60 P eb750b23c51c44e6a5c9e9d6f70e59b0: Deleting consensus metadata
I20260807 02:56:08.529129 8137 ts_tablet_manager.cc:1918] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:08.529304 8137 log.cc:1196] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/0ec7d33b5c9149ef840f225ba409ef60
I20260807 02:56:08.529525 7389 catalog_manager.cc:4841] TS eb750b23c51c44e6a5c9e9d6f70e59b0 (127.1.121.67:45415): tablet 0ec7d33b5c9149ef840f225ba409ef60 (table test-table [id=e1c0b436d2004e56990040e7c258fd6f]) successfully deleted
I20260807 02:56:08.529927 8137 ts_tablet_manager.cc:1939] T 0ec7d33b5c9149ef840f225ba409ef60 P 723ebbff80d44899b9dcb84b19eb7e9a: Deleting consensus metadata
I20260807 02:56:08.531023 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.501457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.1.121.65:44767)
0807 02:56:08.501724 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.501734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.507416 (+ 5682us) server_negotiation.cc:407] Connection header received
0807 02:56:08.507747 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.507756 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.507879 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.507968 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:08.509949 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.511159 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.513399 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.514161 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.514263 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.522160 (+ 7897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:08.522176 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:08.522199 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:08.522251 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:08.526455 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.527658 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.527671 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.527678 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.527864 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.528372 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.528377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.528380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.528871 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:08.529547 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.530333 (+ 786us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.530606 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 02:56:08.531733 7392 catalog_manager.cc:4841] TS 723ebbff80d44899b9dcb84b19eb7e9a (127.1.121.65:44767): tablet 0ec7d33b5c9149ef840f225ba409ef60 (table test-table [id=e1c0b436d2004e56990040e7c258fd6f]) successfully deleted
I20260807 02:56:08.532935 7672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:08.634407 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 7500
I20260807 02:56:08.665330 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 7685
Aug 07 02:56:08 dist-test-slave-3h5h krb5kdc[7351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:56:08.680750 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.666107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.1.121.66:37081)
0807 02:56:08.666316 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.666321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.666340 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:56:08.666521 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.666525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.666620 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.666706 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:08.667978 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.668719 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.670142 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.670541 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.670684 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.674976 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:08.674991 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:08.675009 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:08.675045 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:08.678008 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.678681 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.678695 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.678697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.678822 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.679206 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.679217 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.679220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.679569 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:08.679889 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.680187 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.680413 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 02:56:08.681476 7858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:08.775318 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 7685
I20260807 02:56:08.800091 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 7873
Aug 07 02:56:08 dist-test-slave-3h5h krb5kdc[7351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:56:08.815227 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:08.801143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.1.121.67:45415)
0807 02:56:08.801325 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:08.801329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:08.801346 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:56:08.801405 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:08.801409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:08.801472 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:08.801588 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:08.802862 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.803602 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.804798 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:08.805347 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:08.805443 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:08.809789 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:08.809804 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:08.809822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:08.809865 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:08.812681 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.813326 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.813333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.813335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.813470 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:08.813819 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:08.813823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:08.813825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:08.814179 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:08.814396 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:08.814679 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 02:56:08.814855 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:56:08.815985 8070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:08.906980 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 7873
I20260807 02:56:08.935003 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 7368
I20260807 02:56:08.936301 7454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:09.051028 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 7368
2026-08-07T02:56:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4604 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T02:56:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:09Z Disabled control of system clock
I20260807 02:56:09.136965 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33749
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:33749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 02:56:09.508283 8148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:09.508589 8148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:09.508675 8148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:09.516073 8148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:09.516191 8148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:09.516230 8148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:09.516281 8148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:09.516328 8148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:09.525817 8148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:33749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:09.527139 8148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:09.529134 8148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:09.541590 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:09.541585 8157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:09.541621 8161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:09.542634 8148 server_base.cc:1034] running on GCE node
I20260807 02:56:09.543443 8148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:09.544945 8148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:09.546190 8148 hybrid_clock.cc:648] HybridClock initialized: now 1786071369546170 us; error 68 us; skew 500 ppm
I20260807 02:56:09.546557 8148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:56:09.548913 8148 webserver.cc:460] Webserver started at http://127.1.121.126:37341/ using document root <none> and password file <none>
I20260807 02:56:09.549623 8148 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:09.549772 8148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:09.550122 8148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:09.553172 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c"
format_stamp: "Formatted at 2026-08-07 02:56:09 on dist-test-slave-3h5h"
server_key: "a5402285bc0a9025a49fd3ce11ee2fcb"
server_key_iv: "f180bb4fc36b26e703351e749429a890"
server_key_version: "encryptionkey@0"
I20260807 02:56:09.554008 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c"
format_stamp: "Formatted at 2026-08-07 02:56:09 on dist-test-slave-3h5h"
server_key: "a5402285bc0a9025a49fd3ce11ee2fcb"
server_key_iv: "f180bb4fc36b26e703351e749429a890"
server_key_version: "encryptionkey@0"
I20260807 02:56:09.559180 8148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 02:56:09.562672 8170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:09.563977 8148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:56:09.564175 8148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c"
format_stamp: "Formatted at 2026-08-07 02:56:09 on dist-test-slave-3h5h"
server_key: "a5402285bc0a9025a49fd3ce11ee2fcb"
server_key_iv: "f180bb4fc36b26e703351e749429a890"
server_key_version: "encryptionkey@0"
I20260807 02:56:09.564402 8148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:09.570194 8148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:09.570966 8148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:09.571280 8148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:09.590718 8148 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:33749
I20260807 02:56:09.590811 8249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:33749 every 8 connection(s)
I20260807 02:56:09.592164 8148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:09.596431 8251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:09.600211 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 8148
I20260807 02:56:09.600422 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:09.601151 1509 external_mini_cluster.cc:1428] Setting key 8f6a08af9620ba0f8eb5f9e43bc405e1
I20260807 02:56:09.609902 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:09.603422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.1.121.126:33749)
0807 02:56:09.604485 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:09.604496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:09.604537 (+ 41us) server_negotiation.cc:407] Connection header received
0807 02:56:09.606219 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:09.606243 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:09.606651 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:09.607056 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:09.608165 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:09.608172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:09.608185 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:09.608199 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:09.608265 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:09.608282 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:09.608542 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:09.608686 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 02:56:09.608982 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":422,"threads_started":1}
I20260807 02:56:09.611047 8251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c: Bootstrap starting.
I20260807 02:56:09.615119 8251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:09.616323 8251 log.cc:824] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:09.619594 8251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c: No bootstrap required, opened a new log
I20260807 02:56:09.628281 8251 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33749 } }
I20260807 02:56:09.628662 8251 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:09.628763 8251 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1108c8d32e4d919cd33c48b0e4ea6c, State: Initialized, Role: FOLLOWER
I20260807 02:56:09.629359 8251 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33749 } }
I20260807 02:56:09.629608 8251 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:09.629739 8251 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:09.629925 8251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:09.632104 8251 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33749 } }
I20260807 02:56:09.632630 8251 leader_election.cc:302] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc1108c8d32e4d919cd33c48b0e4ea6c; no voters:
I20260807 02:56:09.633237 8251 leader_election.cc:288] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:09.633394 8258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:09.634235 8258 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [term 1 LEADER]: Becoming Leader. State: Replica: dc1108c8d32e4d919cd33c48b0e4ea6c, State: Running, Role: LEADER
I20260807 02:56:09.634868 8258 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33749 } }
I20260807 02:56:09.635661 8251 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:09.638315 8260 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33749 } } }
I20260807 02:56:09.638998 8260 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:09.638322 8261 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc1108c8d32e4d919cd33c48b0e4ea6c. Latest consensus state: current_term: 1 leader_uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1108c8d32e4d919cd33c48b0e4ea6c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33749 } } }
I20260807 02:56:09.639245 8261 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:09.641932 8269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:09.648020 8269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:09.657756 8269 catalog_manager.cc:1347] Generated new cluster ID: ad02d8d7be624a058f9f87eae6370c20
I20260807 02:56:09.657892 8269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:09.699616 8269 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:09.700086 8269 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:09.712545 8269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c: Generated new TSK 0
I20260807 02:56:09.713465 8269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:56:09.736409 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33749
--builtin_ntp_servers=127.1.121.84:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 02:56:10.100013 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:10.100329 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:10.100497 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:10.108031 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:10.108150 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:10.108330 8283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:56:10.118162 8283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:10.119560 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:10.121471 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:10.135076 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:10.135186 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:10.135152 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:10.135983 8283 server_base.cc:1034] running on GCE node
I20260807 02:56:10.136744 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:10.137801 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:10.139099 8283 hybrid_clock.cc:648] HybridClock initialized: now 1786071370139061 us; error 91 us; skew 500 ppm
I20260807 02:56:10.139477 8283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:56:10.141916 8283 webserver.cc:460] Webserver started at http://127.1.121.65:33369/ using document root <none> and password file <none>
I20260807 02:56:10.142658 8283 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:10.142802 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:10.143152 8283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:10.146517 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "9967dc5c9f664e9d8b98323ea29646a7"
format_stamp: "Formatted at 2026-08-07 02:56:10 on dist-test-slave-3h5h"
server_key: "86444b993cb255e6c147bdc8677c620e"
server_key_iv: "349dbb5e4616d4cb9e868f0984287645"
server_key_version: "encryptionkey@0"
I20260807 02:56:10.147369 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "9967dc5c9f664e9d8b98323ea29646a7"
format_stamp: "Formatted at 2026-08-07 02:56:10 on dist-test-slave-3h5h"
server_key: "86444b993cb255e6c147bdc8677c620e"
server_key_iv: "349dbb5e4616d4cb9e868f0984287645"
server_key_version: "encryptionkey@0"
I20260807 02:56:10.152910 8283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 02:56:10.156478 8299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:10.158051 8283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 02:56:10.158267 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "9967dc5c9f664e9d8b98323ea29646a7"
format_stamp: "Formatted at 2026-08-07 02:56:10 on dist-test-slave-3h5h"
server_key: "86444b993cb255e6c147bdc8677c620e"
server_key_iv: "349dbb5e4616d4cb9e868f0984287645"
server_key_version: "encryptionkey@0"
I20260807 02:56:10.158514 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:10.164553 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:10.165462 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:10.165747 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:10.166772 8283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:10.168373 8283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:10.168507 8283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:10.168658 8283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:10.168749 8283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:10.203282 8283 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:38891
I20260807 02:56:10.203331 8462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:38891 every 8 connection(s)
I20260807 02:56:10.204723 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:56:10.205994 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 8283
I20260807 02:56:10.206193 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:56:10.207078 1509 external_mini_cluster.cc:1428] Setting key ac6e61b316987fcceb6d97e24d564824
I20260807 02:56:10.210726 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33749
--builtin_ntp_servers=127.1.121.84:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 02:56:10.215176 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:10.207881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:41847 (local address 127.1.121.126:33749)
0807 02:56:10.208711 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:10.208716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:10.209881 (+ 1165us) server_negotiation.cc:407] Connection header received
0807 02:56:10.212669 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:10.212674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:10.212757 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:10.212928 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:10.214360 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:10.214367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:10.214371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:10.214391 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:10.214443 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:10.214455 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:10.214603 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:10.214790 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 02:56:10.214864 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":617,"threads_started":1}
I20260807 02:56:10.216109 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:10.208507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33749 (local address 127.1.121.65:41847)
0807 02:56:10.209695 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:10.209745 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:10.212321 (+ 2576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:10.213177 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:10.213185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:10.213277 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:10.214022 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:56:10.214024 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:10.214069 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:56:10.214182 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:10.214556 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:10.214567 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:10.214577 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 02:56:10.214873 (+ 296us) client_negotiation.cc:240] Negotiation successful
0807 02:56:10.215181 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":372,"threads_started":1}
I20260807 02:56:10.217700 8463 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33749
I20260807 02:56:10.218027 8463 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:10.218662 8463 heartbeater.cc:511] Master 127.1.121.126:33749 requested a full tablet report, sending...
I20260807 02:56:10.220788 8195 ts_manager.cc:194] Registered new tserver with Master: 9967dc5c9f664e9d8b98323ea29646a7 (127.1.121.65:38891)
W20260807 02:56:10.553197 8471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:10.553486 8471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:10.553660 8471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:10.561038 8471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:10.561141 8471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:10.561336 8471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:56:10.570684 8471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:10.572055 8471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:10.574023 8471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:10.587687 8479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:10.587756 8481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:10.587720 8483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:10.588620 8471 server_base.cc:1034] running on GCE node
I20260807 02:56:10.589359 8471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:10.590390 8471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:10.591622 8471 hybrid_clock.cc:648] HybridClock initialized: now 1786071370591560 us; error 54 us; skew 500 ppm
I20260807 02:56:10.591952 8471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:56:10.594385 8471 webserver.cc:460] Webserver started at http://127.1.121.66:35249/ using document root <none> and password file <none>
I20260807 02:56:10.595136 8471 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:10.595270 8471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:10.595607 8471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:10.598554 8471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "0c52487128f34dbe8282350af16071f0"
format_stamp: "Formatted at 2026-08-07 02:56:10 on dist-test-slave-3h5h"
server_key: "ec130baad5816389b2e4ecad89139ffe"
server_key_iv: "411ff51af8a598461311fe40d7dcd0cb"
server_key_version: "encryptionkey@0"
I20260807 02:56:10.599359 8471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "0c52487128f34dbe8282350af16071f0"
format_stamp: "Formatted at 2026-08-07 02:56:10 on dist-test-slave-3h5h"
server_key: "ec130baad5816389b2e4ecad89139ffe"
server_key_iv: "411ff51af8a598461311fe40d7dcd0cb"
server_key_version: "encryptionkey@0"
I20260807 02:56:10.604583 8471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 02:56:10.608042 8492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:10.609412 8471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:10.609586 8471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "0c52487128f34dbe8282350af16071f0"
format_stamp: "Formatted at 2026-08-07 02:56:10 on dist-test-slave-3h5h"
server_key: "ec130baad5816389b2e4ecad89139ffe"
server_key_iv: "411ff51af8a598461311fe40d7dcd0cb"
server_key_version: "encryptionkey@0"
I20260807 02:56:10.609826 8471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:10.615445 8471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:10.616246 8471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:10.616537 8471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:10.617525 8471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:10.618937 8471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:10.619081 8471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:10.619222 8471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:10.619313 8471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:10.654838 8471 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:45701
I20260807 02:56:10.654951 8658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:45701 every 8 connection(s)
I20260807 02:56:10.656242 8471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:56:10.664604 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:10.658902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:43303 (local address 127.1.121.126:33749)
0807 02:56:10.659124 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:10.659130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:10.660549 (+ 1419us) server_negotiation.cc:407] Connection header received
0807 02:56:10.662554 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:10.662559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:10.662685 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:10.662813 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:10.663973 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:10.663978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:10.663981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:10.663999 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:10.664031 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:10.664043 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:10.664153 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:10.664319 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 02:56:10.664372 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 02:56:10.665529 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:10.659440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33749 (local address 127.1.121.66:43303)
0807 02:56:10.660385 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:10.660420 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:10.662305 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:10.662991 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:10.662999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:10.663095 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:10.663694 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:56:10.663696 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:10.663733 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:56:10.663813 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:10.664141 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:10.664149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:10.664154 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 02:56:10.664368 (+ 214us) client_negotiation.cc:240] Negotiation successful
0807 02:56:10.664551 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":312,"threads_started":1}
I20260807 02:56:10.666272 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 8471
I20260807 02:56:10.666476 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:56:10.666965 1509 external_mini_cluster.cc:1428] Setting key c6392180ffab49a398cec687a339b5d4
I20260807 02:56:10.667193 8659 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33749
I20260807 02:56:10.667505 8659 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:10.668269 8659 heartbeater.cc:511] Master 127.1.121.126:33749 requested a full tablet report, sending...
I20260807 02:56:10.669836 8195 ts_manager.cc:194] Registered new tserver with Master: 0c52487128f34dbe8282350af16071f0 (127.1.121.66:45701)
I20260807 02:56:10.670105 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33749
--builtin_ntp_servers=127.1.121.84:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 02:56:11.026620 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:11.026914 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:11.027109 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:11.034682 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:11.034785 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:11.034991 8665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:56:11.044776 8665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:11.046262 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:11.048059 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:11.061986 8673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:11.062223 8672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:11.061862 8675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:11.066176 8665 server_base.cc:1034] running on GCE node
I20260807 02:56:11.066835 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:11.068011 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:11.069289 8665 hybrid_clock.cc:648] HybridClock initialized: now 1786071371069180 us; error 70 us; skew 500 ppm
I20260807 02:56:11.069630 8665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:56:11.072096 8665 webserver.cc:460] Webserver started at http://127.1.121.67:43821/ using document root <none> and password file <none>
I20260807 02:56:11.072886 8665 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:11.073033 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:11.073412 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:11.076660 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "1d726a662a44467c90b49f6864c187ae"
format_stamp: "Formatted at 2026-08-07 02:56:11 on dist-test-slave-3h5h"
server_key: "4a944d183f51b60775cc8b98dcafc260"
server_key_iv: "f322317677b0516292246ffb1fd3345f"
server_key_version: "encryptionkey@0"
I20260807 02:56:11.077539 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "1d726a662a44467c90b49f6864c187ae"
format_stamp: "Formatted at 2026-08-07 02:56:11 on dist-test-slave-3h5h"
server_key: "4a944d183f51b60775cc8b98dcafc260"
server_key_iv: "f322317677b0516292246ffb1fd3345f"
server_key_version: "encryptionkey@0"
I20260807 02:56:11.082897 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:56:11.086547 8687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:11.087945 8665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:56:11.088181 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "1d726a662a44467c90b49f6864c187ae"
format_stamp: "Formatted at 2026-08-07 02:56:11 on dist-test-slave-3h5h"
server_key: "4a944d183f51b60775cc8b98dcafc260"
server_key_iv: "f322317677b0516292246ffb1fd3345f"
server_key_version: "encryptionkey@0"
I20260807 02:56:11.088428 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:11.094424 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:11.095224 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:11.095498 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:11.096443 8665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:11.097904 8665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:11.098038 8665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:11.098174 8665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:11.098271 8665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:11.133597 8665 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:41665
I20260807 02:56:11.133706 8854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:41665 every 8 connection(s)
I20260807 02:56:11.135072 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:56:11.136790 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 8665
I20260807 02:56:11.136986 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:56:11.137596 1509 external_mini_cluster.cc:1428] Setting key 60be6732157b9c2d5fe6a1b2f685e84a
I20260807 02:56:11.144716 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.138274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:33013 (local address 127.1.121.126:33749)
0807 02:56:11.138513 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.138519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.140258 (+ 1739us) server_negotiation.cc:407] Connection header received
0807 02:56:11.142249 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.142255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.142335 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.142460 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.143785 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.143791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.143796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.143815 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.143860 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.143872 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.144005 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.144328 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.144403 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 02:56:11.145730 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.139003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33749 (local address 127.1.121.67:33013)
0807 02:56:11.140105 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.140144 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:11.141962 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:11.142709 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:11.142718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:11.142809 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:11.143471 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:56:11.143473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:11.143520 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:56:11.143611 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:11.143980 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:11.144102 (+ 122us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:11.144110 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 02:56:11.144405 (+ 295us) client_negotiation.cc:240] Negotiation successful
0807 02:56:11.144703 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":318,"threads_started":1}
I20260807 02:56:11.147320 8855 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33749
I20260807 02:56:11.147616 8855 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:11.148211 8855 heartbeater.cc:511] Master 127.1.121.126:33749 requested a full tablet report, sending...
I20260807 02:56:11.150233 8195 ts_manager.cc:194] Registered new tserver with Master: 1d726a662a44467c90b49f6864c187ae (127.1.121.67:41665)
I20260807 02:56:11.153543 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:56:11.161113 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.158295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.1.121.126:33749)
0807 02:56:11.158473 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.158478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.159046 (+ 568us) server_negotiation.cc:407] Connection header received
0807 02:56:11.159242 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.159247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.159310 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.159415 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.160353 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.160358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.160361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.160379 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.160416 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.160428 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.160545 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.160639 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.160690 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 02:56:11.165407 8195 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:56:11.173249 8195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:56:11.202983 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.187385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.1.121.66:45701)
0807 02:56:11.199374 (+ 11989us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.199383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.199415 (+ 32us) server_negotiation.cc:407] Connection header received
0807 02:56:11.199490 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.199500 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.199709 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.200363 (+ 654us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.201662 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.201671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.201683 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.201702 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.201762 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.201785 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.202468 (+ 683us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.202547 (+ 79us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.202651 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11804,"thread_start_us":13107,"threads_started":1}
I20260807 02:56:11.203678 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.187191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:45701 (local address 127.0.0.1:40334)
0807 02:56:11.198745 (+ 11554us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.198766 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:11.198856 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:11.199899 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:11.199904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:11.199925 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:11.201397 (+ 1472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:56:11.201401 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:11.201416 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:56:11.201459 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:11.201909 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:11.201914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:11.201918 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 02:56:11.202054 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 02:56:11.202102 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11419,"thread_start_us":11289,"threads_started":1}
I20260807 02:56:11.204241 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.187017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:41665 (local address 127.0.0.1:46808)
0807 02:56:11.198012 (+ 10995us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.198062 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:11.198203 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:11.199054 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:11.199063 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:11.199163 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:11.200124 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:56:11.200128 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:11.200165 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:56:11.200264 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:11.201629 (+ 1365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:11.201643 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:11.201677 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 02:56:11.201853 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 02:56:11.201947 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10734,"thread_start_us":10618,"threads_started":1}
I20260807 02:56:11.205353 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.187419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.1.121.67:41665)
0807 02:56:11.197621 (+ 10202us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.197632 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.198349 (+ 717us) server_negotiation.cc:407] Connection header received
0807 02:56:11.198449 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.198472 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.198833 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.199686 (+ 853us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.201372 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.201380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.201395 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.201413 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.201480 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.201504 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.204755 (+ 3251us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.204818 (+ 63us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.204935 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9993,"thread_start_us":9805,"threads_started":1}
I20260807 02:56:11.209045 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.200936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.0.0.1:56262)
0807 02:56:11.203897 (+ 2961us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.203918 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:11.204009 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:11.206716 (+ 2707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:11.206723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:11.206759 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:11.207300 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:56:11.207303 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:11.207319 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:56:11.207361 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:11.208629 (+ 1268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:11.208634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:11.208640 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 02:56:11.208721 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 02:56:11.208767 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2862,"thread_start_us":4786,"threads_started":1}
I20260807 02:56:11.209821 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.199267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.1.121.65:38891)
0807 02:56:11.206205 (+ 6938us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.206213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.206242 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:56:11.206308 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.206315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.206529 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.207029 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.208398 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.208406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.208419 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.208441 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.208489 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.208506 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.209299 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.209363 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.209450 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6771,"thread_start_us":6651,"threads_started":1}
I20260807 02:56:11.209611 8562 tablet_service.cc:1467] Processing CreateTablet for tablet 24602288233148a4ad808d7646583079 (DEFAULT_TABLE table=test-table [id=94d458ac46ee4f1ca2f7c002244bc1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:11.209611 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 24602288233148a4ad808d7646583079 (DEFAULT_TABLE table=test-table [id=94d458ac46ee4f1ca2f7c002244bc1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:11.211158 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24602288233148a4ad808d7646583079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:11.212131 8562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24602288233148a4ad808d7646583079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:11.213646 8375 tablet_service.cc:1467] Processing CreateTablet for tablet 24602288233148a4ad808d7646583079 (DEFAULT_TABLE table=test-table [id=94d458ac46ee4f1ca2f7c002244bc1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:11.214879 8375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24602288233148a4ad808d7646583079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:11.223737 8463 heartbeater.cc:503] Master 127.1.121.126:33749 was elected leader, sending a full tablet report...
I20260807 02:56:11.228018 8887 tablet_bootstrap.cc:492] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: Bootstrap starting.
I20260807 02:56:11.228760 8892 tablet_bootstrap.cc:492] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: Bootstrap starting.
I20260807 02:56:11.230176 8885 tablet_bootstrap.cc:492] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7: Bootstrap starting.
I20260807 02:56:11.232066 8887 tablet_bootstrap.cc:654] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:11.232343 8892 tablet_bootstrap.cc:654] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:11.233210 8887 log.cc:824] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:11.233615 8892 log.cc:824] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:11.233865 8885 tablet_bootstrap.cc:654] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:11.234965 8885 log.cc:824] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:11.236506 8892 tablet_bootstrap.cc:492] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: No bootstrap required, opened a new log
I20260807 02:56:11.236806 8892 ts_tablet_manager.cc:1397] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260807 02:56:11.236966 8887 tablet_bootstrap.cc:492] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: No bootstrap required, opened a new log
I20260807 02:56:11.237238 8887 ts_tablet_manager.cc:1397] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260807 02:56:11.239730 8885 tablet_bootstrap.cc:492] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7: No bootstrap required, opened a new log
I20260807 02:56:11.239990 8885 ts_tablet_manager.cc:1397] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260807 02:56:11.244442 8887 raft_consensus.cc:348] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } }
I20260807 02:56:11.244892 8887 raft_consensus.cc:374] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:11.245007 8887 raft_consensus.cc:729] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d726a662a44467c90b49f6864c187ae, State: Initialized, Role: FOLLOWER
I20260807 02:56:11.245697 8887 consensus_queue.cc:260] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } }
I20260807 02:56:11.246884 8855 heartbeater.cc:503] Master 127.1.121.126:33749 was elected leader, sending a full tablet report...
I20260807 02:56:11.247577 8887 ts_tablet_manager.cc:1428] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260807 02:56:11.247658 8892 raft_consensus.cc:348] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } }
I20260807 02:56:11.248083 8892 raft_consensus.cc:374] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:11.248183 8892 raft_consensus.cc:729] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c52487128f34dbe8282350af16071f0, State: Initialized, Role: FOLLOWER
I20260807 02:56:11.248781 8892 consensus_queue.cc:260] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } }
I20260807 02:56:11.249960 8659 heartbeater.cc:503] Master 127.1.121.126:33749 was elected leader, sending a full tablet report...
I20260807 02:56:11.250564 8892 ts_tablet_manager.cc:1428] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260807 02:56:11.250548 8885 raft_consensus.cc:348] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } }
I20260807 02:56:11.251004 8885 raft_consensus.cc:374] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:11.251102 8885 raft_consensus.cc:729] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9967dc5c9f664e9d8b98323ea29646a7, State: Initialized, Role: FOLLOWER
I20260807 02:56:11.251742 8885 consensus_queue.cc:260] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } }
I20260807 02:56:11.254431 8885 ts_tablet_manager.cc:1428] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260807 02:56:11.305370 8897 raft_consensus.cc:482] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:11.305768 8897 raft_consensus.cc:504] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } }
I20260807 02:56:11.307780 8897 leader_election.cc:288] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c52487128f34dbe8282350af16071f0 (127.1.121.66:45701), 9967dc5c9f664e9d8b98323ea29646a7 (127.1.121.65:38891)
I20260807 02:56:11.310699 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.308074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:45701 (local address 127.1.121.67:59355)
0807 02:56:11.308295 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.308316 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:11.308434 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:11.309130 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:11.309135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:11.309170 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:11.309831 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:56:11.309833 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:11.309849 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:56:11.309898 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:11.310253 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:11.310258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:11.310262 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 02:56:11.310364 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 02:56:11.310426 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 02:56:11.310699 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.308321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:59355 (local address 127.1.121.66:45701)
0807 02:56:11.308535 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.308543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.308565 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:56:11.308641 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.308649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.308730 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.309109 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.310062 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.310068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.310073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.310091 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.310131 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.310144 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.310266 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.310360 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.310431 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 02:56:11.311419 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.309256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:57125 (local address 127.1.121.65:38891)
0807 02:56:11.309446 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.309451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.309467 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:56:11.309525 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.309529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.309569 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.309685 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.310690 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.310695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.310699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.310724 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.310775 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.310793 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.310926 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.311105 (+ 179us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.311169 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 02:56:11.311533 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.308486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.1.121.67:57125)
0807 02:56:11.309188 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.309211 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:11.309309 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:11.309704 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:11.309709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:11.309733 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:11.310477 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:56:11.310480 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:11.310493 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:56:11.310528 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:11.310976 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:11.310981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:11.310985 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 02:56:11.311090 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 02:56:11.311141 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":317,"threads_started":1}
I20260807 02:56:11.311867 8594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24602288233148a4ad808d7646583079" candidate_uuid: "1d726a662a44467c90b49f6864c187ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c52487128f34dbe8282350af16071f0" is_pre_election: true
I20260807 02:56:11.312505 8594 raft_consensus.cc:2454] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d726a662a44467c90b49f6864c187ae in term 0.
I20260807 02:56:11.313784 8690 leader_election.cc:302] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c52487128f34dbe8282350af16071f0, 1d726a662a44467c90b49f6864c187ae; no voters:
I20260807 02:56:11.313828 8401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24602288233148a4ad808d7646583079" candidate_uuid: "1d726a662a44467c90b49f6864c187ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9967dc5c9f664e9d8b98323ea29646a7" is_pre_election: true
I20260807 02:56:11.314270 8897 raft_consensus.cc:2781] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:56:11.314428 8897 raft_consensus.cc:482] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:56:11.314419 8401 raft_consensus.cc:2454] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d726a662a44467c90b49f6864c187ae in term 0.
I20260807 02:56:11.314565 8897 raft_consensus.cc:3037] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:11.316995 8897 raft_consensus.cc:504] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } }
I20260807 02:56:11.317601 8897 leader_election.cc:288] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [CANDIDATE]: Term 1 election: Requested vote from peers 0c52487128f34dbe8282350af16071f0 (127.1.121.66:45701), 9967dc5c9f664e9d8b98323ea29646a7 (127.1.121.65:38891)
I20260807 02:56:11.318266 8594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24602288233148a4ad808d7646583079" candidate_uuid: "1d726a662a44467c90b49f6864c187ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c52487128f34dbe8282350af16071f0"
I20260807 02:56:11.318312 8401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24602288233148a4ad808d7646583079" candidate_uuid: "1d726a662a44467c90b49f6864c187ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9967dc5c9f664e9d8b98323ea29646a7"
I20260807 02:56:11.318506 8594 raft_consensus.cc:3037] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:11.318506 8401 raft_consensus.cc:3037] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:11.321947 8401 raft_consensus.cc:2454] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d726a662a44467c90b49f6864c187ae in term 1.
I20260807 02:56:11.321947 8594 raft_consensus.cc:2454] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d726a662a44467c90b49f6864c187ae in term 1.
I20260807 02:56:11.322376 8690 leader_election.cc:302] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d726a662a44467c90b49f6864c187ae, 9967dc5c9f664e9d8b98323ea29646a7; no voters:
I20260807 02:56:11.322633 8897 raft_consensus.cc:2781] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:11.323133 8897 raft_consensus.cc:686] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 1 LEADER]: Becoming Leader. State: Replica: 1d726a662a44467c90b49f6864c187ae, State: Running, Role: LEADER
I20260807 02:56:11.323616 8897 consensus_queue.cc:237] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } }
I20260807 02:56:11.328112 8191 catalog_manager.cc:5495] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d726a662a44467c90b49f6864c187ae (127.1.121.67). New cstate: current_term: 1 leader_uuid: "1d726a662a44467c90b49f6864c187ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d726a662a44467c90b49f6864c187ae" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 41665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 } health_report { overall_health: UNKNOWN } } }
I20260807 02:56:11.354410 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.352423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.1.121.67:41665)
0807 02:56:11.352620 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.352625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.352644 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:56:11.352781 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.352785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.352924 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.353020 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.353923 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.353927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.353930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.353943 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.353971 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.353983 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.354083 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.354169 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.354217 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:56:11.363253 8594 raft_consensus.cc:1261] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [term 1 FOLLOWER]: Refusing update from remote peer 1d726a662a44467c90b49f6864c187ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:11.363312 8401 raft_consensus.cc:1261] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [term 1 FOLLOWER]: Refusing update from remote peer 1d726a662a44467c90b49f6864c187ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:11.364228 8903 consensus_queue.cc:1035] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c52487128f34dbe8282350af16071f0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 45701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:11.364617 8897 consensus_queue.cc:1035] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9967dc5c9f664e9d8b98323ea29646a7" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:11.393324 8195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:11.393623 8195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:11.397456 8195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc1108c8d32e4d919cd33c48b0e4ea6c: Sending DeleteTablet for 3 replicas of tablet 24602288233148a4ad808d7646583079
I20260807 02:56:11.398572 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet 24602288233148a4ad808d7646583079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:11 UTC) from {username='slave'} at 127.0.0.1:46808
I20260807 02:56:11.398871 8562 tablet_service.cc:1514] Processing DeleteTablet for tablet 24602288233148a4ad808d7646583079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:11 UTC) from {username='slave'} at 127.0.0.1:40334
I20260807 02:56:11.399091 8375 tablet_service.cc:1514] Processing DeleteTablet for tablet 24602288233148a4ad808d7646583079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:11 UTC) from {username='slave'} at 127.0.0.1:56262
I20260807 02:56:11.401051 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 8283
I20260807 02:56:11.409940 8925 tablet_replica.cc:330] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7: stopping tablet replica
I20260807 02:56:11.410549 8925 raft_consensus.cc:2229] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:11.411201 8925 raft_consensus.cc:2258] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:11.411868 8923 tablet_replica.cc:330] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: stopping tablet replica
I20260807 02:56:11.412276 8923 raft_consensus.cc:2229] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:56:11.413040 8923 raft_consensus.cc:2258] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:11.415562 8925 ts_tablet_manager.cc:1905] T 24602288233148a4ad808d7646583079 P 9967dc5c9f664e9d8b98323ea29646a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:11.417402 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.402700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.1.121.65:38891)
0807 02:56:11.402929 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.402938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.405403 (+ 2465us) server_negotiation.cc:407] Connection header received
0807 02:56:11.414208 (+ 8805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.414218 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.414343 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.414481 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.416060 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.416069 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.416082 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.416110 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.416158 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.416179 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.416338 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.416882 (+ 544us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.416966 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 02:56:11.418252 8457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:11.419247 8924 tablet_replica.cc:330] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: stopping tablet replica
I20260807 02:56:11.419739 8924 raft_consensus.cc:2229] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:11.420337 8924 raft_consensus.cc:2258] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:11.422108 8923 ts_tablet_manager.cc:1905] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:11.422684 8924 ts_tablet_manager.cc:1905] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:11.429803 8923 ts_tablet_manager.cc:1918] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:11.429997 8923 log.cc:1196] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/24602288233148a4ad808d7646583079
I20260807 02:56:11.430598 8923 ts_tablet_manager.cc:1939] T 24602288233148a4ad808d7646583079 P 1d726a662a44467c90b49f6864c187ae: Deleting consensus metadata
I20260807 02:56:11.432036 8924 ts_tablet_manager.cc:1918] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:11.432238 8924 log.cc:1196] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/24602288233148a4ad808d7646583079
I20260807 02:56:11.432579 8176 catalog_manager.cc:4841] TS 1d726a662a44467c90b49f6864c187ae (127.1.121.67:41665): tablet 24602288233148a4ad808d7646583079 (table test-table [id=94d458ac46ee4f1ca2f7c002244bc1d3]) successfully deleted
I20260807 02:56:11.432900 8924 ts_tablet_manager.cc:1939] T 24602288233148a4ad808d7646583079 P 0c52487128f34dbe8282350af16071f0: Deleting consensus metadata
I20260807 02:56:11.436189 8177 catalog_manager.cc:4841] TS 0c52487128f34dbe8282350af16071f0 (127.1.121.66:45701): tablet 24602288233148a4ad808d7646583079 (table test-table [id=94d458ac46ee4f1ca2f7c002244bc1d3]) successfully deleted
I20260807 02:56:11.525556 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 8283
I20260807 02:56:11.554747 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 8471
W20260807 02:56:11.554695 8177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.121.65:38891 (error 108)
I20260807 02:56:11.557405 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.556794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.0.0.1:56280)
0807 02:56:11.556990 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.557194 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260807 02:56:11.557682 8177 catalog_manager.cc:4584] TS 9967dc5c9f664e9d8b98323ea29646a7 (127.1.121.65:38891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24602288233148a4ad808d7646583079: Network error: Client connection negotiation failed: client connection to 127.1.121.65:38891: connect: Connection refused (error 111)
I20260807 02:56:11.558635 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.555716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.1.121.66:45701)
0807 02:56:11.556340 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.556347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.556365 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:56:11.556431 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.556437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.556548 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.556665 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.557853 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.557859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.557864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.557885 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.557925 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.557941 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.558058 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.558265 (+ 207us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.558318 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488}
I20260807 02:56:11.559386 8653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:11.611770 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.611144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.0.0.1:56288)
0807 02:56:11.611461 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.611572 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260807 02:56:11.613178 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.612675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.0.0.1:56302)
0807 02:56:11.612926 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.613012 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260807 02:56:11.665246 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 8471
I20260807 02:56:11.674746 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.674103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.0.0.1:56306)
0807 02:56:11.674402 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.674552 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 02:56:11.676088 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.675583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.0.0.1:56316)
0807 02:56:11.675812 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.675902 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260807 02:56:11.694235 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 8665
I20260807 02:56:11.697505 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.695226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.1.121.67:41665)
0807 02:56:11.695442 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:11.695450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:11.695473 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:56:11.695570 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:11.695576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:11.695642 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:11.695775 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:11.696744 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:11.696751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:11.696756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:11.696775 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:11.696815 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:11.696850 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:11.697017 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:11.697090 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 02:56:11.697152 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 02:56:11.698364 8849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:11.757483 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.756820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.0.0.1:56332)
0807 02:56:11.757122 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.757234 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260807 02:56:11.758941 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.758462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.0.0.1:56338)
0807 02:56:11.758721 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.758807 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260807 02:56:11.822279 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 8665
I20260807 02:56:11.851112 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 8148
I20260807 02:56:11.852502 8245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:11.966922 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:11.965912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38891 (local address 127.0.0.1:56348)
0807 02:56:11.966115 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:11.966272 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 02:56:11.967114 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 8148
2026-08-07T02:56:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2915 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 02:56:12.020462 1509 mini_oidc.cc:152] Starting JWKS server
I20260807 02:56:12.024680 1509 webserver.cc:460] Webserver started at https://127.0.0.1:39027/ using document root <none> and password file <none>
I20260807 02:56:12.025104 1509 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 02:56:12.025985 1509 webserver.cc:460] Webserver started at http://127.0.0.1:45869/ using document root <none> and password file <none>
2026-08-07T02:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:12Z Disabled control of system clock
I20260807 02:56:12.064636 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34211
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:34211
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39027/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/testchainca.cert with env {}
W20260807 02:56:12.402236 8941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:12.402616 8941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:12.402796 8941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:12.410076 8941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:12.410247 8941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:12.410344 8941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:12.410452 8941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:12.410543 8941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 02:56:12.410635 8941 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 02:56:12.410718 8941 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39027/jwks.json
I20260807 02:56:12.420785 8941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:34211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39027/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:12.422199 8941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:12.423547 8941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:12.436247 8947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:12.436292 8953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:12.436208 8948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:12.437170 8941 server_base.cc:1034] running on GCE node
I20260807 02:56:12.437906 8941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:12.439239 8941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:12.440457 8941 hybrid_clock.cc:648] HybridClock initialized: now 1786071372440396 us; error 36 us; skew 500 ppm
I20260807 02:56:12.440934 8941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:56:12.443159 8941 webserver.cc:460] Webserver started at http://127.1.121.126:35339/ using document root <none> and password file <none>
I20260807 02:56:12.443915 8941 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:12.444036 8941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:12.444393 8941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:12.447435 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "d395ad7800fd48049fc0d1958c06d264"
format_stamp: "Formatted at 2026-08-07 02:56:12 on dist-test-slave-3h5h"
server_key: "cba04854529aafcafe2c4bf6abbee766"
server_key_iv: "52382210102cfc704e84fb287259a053"
server_key_version: "encryptionkey@0"
I20260807 02:56:12.448261 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "d395ad7800fd48049fc0d1958c06d264"
format_stamp: "Formatted at 2026-08-07 02:56:12 on dist-test-slave-3h5h"
server_key: "cba04854529aafcafe2c4bf6abbee766"
server_key_iv: "52382210102cfc704e84fb287259a053"
server_key_version: "encryptionkey@0"
I20260807 02:56:12.453528 8941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 02:56:12.457104 8960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:12.458459 8941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:56:12.458643 8941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "d395ad7800fd48049fc0d1958c06d264"
format_stamp: "Formatted at 2026-08-07 02:56:12 on dist-test-slave-3h5h"
server_key: "cba04854529aafcafe2c4bf6abbee766"
server_key_iv: "52382210102cfc704e84fb287259a053"
server_key_version: "encryptionkey@0"
I20260807 02:56:12.458865 8941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:12.483254 8941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:12.484170 8941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:12.484512 8941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:12.503788 8941 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:34211
I20260807 02:56:12.503813 9043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:34211 every 8 connection(s)
I20260807 02:56:12.505242 8941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:12.508496 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 8941
I20260807 02:56:12.508728 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:12.509431 9044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:12.509429 1509 external_mini_cluster.cc:1428] Setting key e18a627e78b085e0d40661dc8194cd4c
I20260807 02:56:12.527561 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:12.511460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.1.121.126:34211)
0807 02:56:12.512717 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:12.512743 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:12.512812 (+ 69us) server_negotiation.cc:407] Connection header received
0807 02:56:12.515566 (+ 2754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:12.515608 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:12.516329 (+ 721us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:12.516885 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:12.517540 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:12.519921 (+ 2381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:12.521474 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:12.522668 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:12.522904 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:12.524245 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:12.524258 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:12.524277 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:12.525214 (+ 937us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:12.525342 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:12.525373 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:12.525524 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:12.525716 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 02:56:12.526084 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":391,"threads_started":1}
I20260807 02:56:12.528277 9044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264: Bootstrap starting.
I20260807 02:56:12.531474 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:56:12.532335 9044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:12.533644 9044 log.cc:824] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:12.536810 9044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264: No bootstrap required, opened a new log
I20260807 02:56:12.544040 9044 raft_consensus.cc:348] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d395ad7800fd48049fc0d1958c06d264" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34211 } }
I20260807 02:56:12.544412 9044 raft_consensus.cc:374] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:12.544513 9044 raft_consensus.cc:729] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d395ad7800fd48049fc0d1958c06d264, State: Initialized, Role: FOLLOWER
I20260807 02:56:12.545138 9044 consensus_queue.cc:260] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d395ad7800fd48049fc0d1958c06d264" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34211 } }
I20260807 02:56:12.545374 9044 raft_consensus.cc:388] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:12.545526 9044 raft_consensus.cc:482] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:12.545691 9044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:12.547932 9044 raft_consensus.cc:504] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d395ad7800fd48049fc0d1958c06d264" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34211 } }
I20260807 02:56:12.548444 9044 leader_election.cc:302] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d395ad7800fd48049fc0d1958c06d264; no voters:
I20260807 02:56:12.549104 9044 leader_election.cc:288] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:12.549296 9050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:12.550421 9050 raft_consensus.cc:686] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [term 1 LEADER]: Becoming Leader. State: Replica: d395ad7800fd48049fc0d1958c06d264, State: Running, Role: LEADER
I20260807 02:56:12.551013 9050 consensus_queue.cc:237] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d395ad7800fd48049fc0d1958c06d264" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34211 } }
I20260807 02:56:12.551311 9044 sys_catalog.cc:564] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:12.554533 9053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d395ad7800fd48049fc0d1958c06d264. Latest consensus state: current_term: 1 leader_uuid: "d395ad7800fd48049fc0d1958c06d264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d395ad7800fd48049fc0d1958c06d264" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34211 } } }
I20260807 02:56:12.554559 9051 sys_catalog.cc:455] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d395ad7800fd48049fc0d1958c06d264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d395ad7800fd48049fc0d1958c06d264" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34211 } } }
I20260807 02:56:12.555050 9053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:12.555055 9051 sys_catalog.cc:458] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:12.555933 9070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:12.560920 9070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:12.569620 9070 catalog_manager.cc:1347] Generated new cluster ID: bf7fec2e72ea49d9b0d941fd0a119432
I20260807 02:56:12.569747 9070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:12.583364 9070 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:12.584700 9070 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:12.591205 9070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d395ad7800fd48049fc0d1958c06d264: Generated new TSK 0
I20260807 02:56:12.592023 9070 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 02:56:12.635527 9092 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 02:56:12.635762 9092 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 02:56:12.638481 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:12.613494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.1.121.126:34211)
0807 02:56:12.613710 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:12.613722 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:12.614318 (+ 596us) server_negotiation.cc:407] Connection header received
0807 02:56:12.614708 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:12.614715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:12.614814 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:12.614992 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0807 02:56:12.615571 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:12.616695 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:12.618221 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:12.618941 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:12.619072 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:12.619649 (+ 577us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 02:56:12.637718 (+ 18069us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 02:56:12.638111 (+ 393us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 02:56:12.638621 9048 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 02:56:12.640079 9089 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:12.613495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:34211 (local address 127.0.0.1:46834)
0807 02:56:12.614210 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:12.614227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:12.614380 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:12.615026 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:12.615030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:12.615040 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 02:56:12.615397 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:12.615406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:12.616890 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:12.616896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:12.618031 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:12.618043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:12.619134 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:12.619139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:12.619290 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:12.619354 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 02:56:12.638039 (+ 18685us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 02:56:12.638370 (+ 331us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.121.126:34211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":364,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.121.126:34211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 02:56:12.642120 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 8941
I20260807 02:56:12.643307 9035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:12.739425 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 8941
2026-08-07T02:56:12Z chronyd exiting
I20260807 02:56:12.797813 1509 test_util.cc:175] -----------------------------------------------
I20260807 02:56:12.797957 1509 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786071323612974-1509-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (782 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 02:56:12.803344 1509 mini_oidc.cc:152] Starting JWKS server
I20260807 02:56:12.805704 1509 webserver.cc:460] Webserver started at https://127.0.0.1:34955/ using document root <none> and password file <none>
I20260807 02:56:12.805966 1509 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 02:56:12.806722 1509 webserver.cc:460] Webserver started at http://127.0.0.1:44633/ using document root <none> and password file <none>
2026-08-07T02:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:12Z Disabled control of system clock
I20260807 02:56:12.845922 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:38849
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:36655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:38849
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34955/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/testchainca.cert with env {}
W20260807 02:56:13.197422 9100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:13.197723 9100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:13.197932 9100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:13.205368 9100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:13.205510 9100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:13.205583 9100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:13.205705 9100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:13.205786 9100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 02:56:13.205878 9100 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 02:56:13.205951 9100 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34955/jwks.json
I20260807 02:56:13.216572 9100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:36655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:38849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:38849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34955/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:13.218050 9100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:13.219398 9100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:13.232087 9108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:13.232028 9109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:13.232146 9111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:13.232868 9100 server_base.cc:1034] running on GCE node
I20260807 02:56:13.233624 9100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:13.235008 9100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:13.236224 9100 hybrid_clock.cc:648] HybridClock initialized: now 1786071373236174 us; error 56 us; skew 500 ppm
I20260807 02:56:13.236611 9100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:56:13.243292 9100 webserver.cc:460] Webserver started at http://127.1.121.126:36297/ using document root <none> and password file <none>
I20260807 02:56:13.244125 9100 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:13.244275 9100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:13.244674 9100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:13.248032 9100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "4421aa1e877b4986abcaaddf3224aa39"
format_stamp: "Formatted at 2026-08-07 02:56:13 on dist-test-slave-3h5h"
server_key: "55e80ec74234e8656035ecced3719068"
server_key_iv: "e601858bdbbbbed50956d91f766636a0"
server_key_version: "encryptionkey@0"
I20260807 02:56:13.248999 9100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "4421aa1e877b4986abcaaddf3224aa39"
format_stamp: "Formatted at 2026-08-07 02:56:13 on dist-test-slave-3h5h"
server_key: "55e80ec74234e8656035ecced3719068"
server_key_iv: "e601858bdbbbbed50956d91f766636a0"
server_key_version: "encryptionkey@0"
I20260807 02:56:13.254781 9100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:56:13.258584 9120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:13.260031 9100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:56:13.260237 9100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "4421aa1e877b4986abcaaddf3224aa39"
format_stamp: "Formatted at 2026-08-07 02:56:13 on dist-test-slave-3h5h"
server_key: "55e80ec74234e8656035ecced3719068"
server_key_iv: "e601858bdbbbbed50956d91f766636a0"
server_key_version: "encryptionkey@0"
I20260807 02:56:13.260474 9100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:13.272583 9100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:13.273466 9100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:13.273842 9100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:13.293156 9100 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:38849
I20260807 02:56:13.293183 9196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:38849 every 8 connection(s)
I20260807 02:56:13.294603 9100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:13.300002 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 9100
I20260807 02:56:13.300209 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:13.301211 1509 external_mini_cluster.cc:1428] Setting key 7fc224ed681ec24f4a1fc6e4f95bba42
I20260807 02:56:13.302034 9197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:13.316875 9197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39: Bootstrap starting.
I20260807 02:56:13.320708 9197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:13.321897 9197 log.cc:824] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:13.321969 9200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:13.309879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.1.121.126:38849)
0807 02:56:13.311129 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:13.311148 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:13.311205 (+ 57us) server_negotiation.cc:407] Connection header received
0807 02:56:13.312810 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:13.312885 (+ 75us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:13.313384 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:13.313749 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:13.314386 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:13.315692 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:13.317517 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:13.318180 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:13.318310 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:13.319693 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:13.319701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:13.319714 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:13.320259 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:13.320352 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:13.320372 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:13.320462 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:13.320719 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 02:56:13.321037 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":391,"threads_started":1}
I20260807 02:56:13.325160 9197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39: No bootstrap required, opened a new log
I20260807 02:56:13.325947 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:56:13.333264 9197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4421aa1e877b4986abcaaddf3224aa39" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38849 } }
I20260807 02:56:13.333667 9197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:13.333799 9197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4421aa1e877b4986abcaaddf3224aa39, State: Initialized, Role: FOLLOWER
I20260807 02:56:13.334371 9197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4421aa1e877b4986abcaaddf3224aa39" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38849 } }
I20260807 02:56:13.334630 9197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:13.334794 9197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:13.334972 9197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:13.337374 9197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4421aa1e877b4986abcaaddf3224aa39" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38849 } }
I20260807 02:56:13.337877 9197 leader_election.cc:302] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4421aa1e877b4986abcaaddf3224aa39; no voters:
I20260807 02:56:13.338460 9197 leader_election.cc:288] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:13.338609 9203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:13.339478 9203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [term 1 LEADER]: Becoming Leader. State: Replica: 4421aa1e877b4986abcaaddf3224aa39, State: Running, Role: LEADER
I20260807 02:56:13.340037 9203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4421aa1e877b4986abcaaddf3224aa39" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38849 } }
I20260807 02:56:13.340698 9197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:13.342893 9204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4421aa1e877b4986abcaaddf3224aa39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4421aa1e877b4986abcaaddf3224aa39" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38849 } } }
I20260807 02:56:13.343338 9204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:13.343904 9206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4421aa1e877b4986abcaaddf3224aa39. Latest consensus state: current_term: 1 leader_uuid: "4421aa1e877b4986abcaaddf3224aa39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4421aa1e877b4986abcaaddf3224aa39" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38849 } } }
I20260807 02:56:13.344277 9206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:13.344326 9215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:13.357808 9215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:13.367566 9215 catalog_manager.cc:1347] Generated new cluster ID: 4a469f63a046473391717c20e53e8222
I20260807 02:56:13.367719 9215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:13.387874 9215 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:13.389828 9215 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:13.399243 9215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4421aa1e877b4986abcaaddf3224aa39: Generated new TSK 0
I20260807 02:56:13.400118 9215 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 02:56:13.440012 9240 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 02:56:13.440171 9240 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 02:56:13.441354 9200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:13.413291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.1.121.126:38849)
0807 02:56:13.413499 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:13.413506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:13.414145 (+ 639us) server_negotiation.cc:407] Connection header received
0807 02:56:13.414401 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:13.414405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:13.414486 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:13.414613 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0807 02:56:13.415140 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:13.415885 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:13.417977 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:13.418418 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:13.418517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:13.419107 (+ 590us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 02:56:13.440516 (+ 21409us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 02:56:13.441007 (+ 491us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33276: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 02:56:13.441491 9200 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33276: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260807 02:56:13.441543 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:13.413253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:38849 (local address 127.0.0.1:33276)
0807 02:56:13.414015 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:13.414035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:13.414213 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:13.414667 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:13.414671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:13.414684 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0807 02:56:13.414982 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:13.414995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:13.416028 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:13.416035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:13.417831 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:13.417847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:13.418609 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:13.418619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:13.418817 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:13.418864 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 02:56:13.440794 (+ 21930us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 02:56:13.441105 (+ 311us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.121.126:38849: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":450,"threads_started":1}
I20260807 02:56:13.443101 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 9100
I20260807 02:56:13.444344 9190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:13.542429 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 9100
2026-08-07T02:56:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (807 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 02:56:13.609194 1509 mini_oidc.cc:152] Starting JWKS server
I20260807 02:56:13.611135 1509 webserver.cc:460] Webserver started at https://127.0.0.1:33351/ using document root <none> and password file <none>
I20260807 02:56:13.611423 1509 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 02:56:13.612082 1509 webserver.cc:460] Webserver started at http://127.0.0.1:38801/ using document root <none> and password file <none>
2026-08-07T02:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:13Z Disabled control of system clock
I20260807 02:56:13.649818 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45951
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:40771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:45951
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33351/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 02:56:14.001446 9254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:14.001755 9254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:14.001935 9254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:14.009272 9254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:14.009399 9254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:14.009483 9254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:14.009594 9254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:14.009657 9254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 02:56:14.009757 9254 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 02:56:14.009804 9254 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33351/jwks.json
I20260807 02:56:14.019400 9254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:45951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33351/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:14.020884 9254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:14.022825 9254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:14.037082 9259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:14.037178 9262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:14.037132 9260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:14.037858 9254 server_base.cc:1034] running on GCE node
I20260807 02:56:14.038712 9254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:14.040230 9254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:14.041460 9254 hybrid_clock.cc:648] HybridClock initialized: now 1786071374041402 us; error 52 us; skew 500 ppm
I20260807 02:56:14.041824 9254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:56:14.044246 9254 webserver.cc:460] Webserver started at http://127.1.121.126:36499/ using document root <none> and password file <none>
I20260807 02:56:14.045077 9254 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:14.045222 9254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:14.045562 9254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:14.048553 9254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "55583e24acd342968c0a9f990002fc7a"
format_stamp: "Formatted at 2026-08-07 02:56:14 on dist-test-slave-3h5h"
server_key: "bf93c3f5ad0125ae175aed8a36dfdcde"
server_key_iv: "291e2473543ec57fcb92fb4267f6d219"
server_key_version: "encryptionkey@0"
I20260807 02:56:14.049490 9254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "55583e24acd342968c0a9f990002fc7a"
format_stamp: "Formatted at 2026-08-07 02:56:14 on dist-test-slave-3h5h"
server_key: "bf93c3f5ad0125ae175aed8a36dfdcde"
server_key_iv: "291e2473543ec57fcb92fb4267f6d219"
server_key_version: "encryptionkey@0"
I20260807 02:56:14.054791 9254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 02:56:14.058312 9271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:14.059693 9254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:56:14.059893 9254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "55583e24acd342968c0a9f990002fc7a"
format_stamp: "Formatted at 2026-08-07 02:56:14 on dist-test-slave-3h5h"
server_key: "bf93c3f5ad0125ae175aed8a36dfdcde"
server_key_iv: "291e2473543ec57fcb92fb4267f6d219"
server_key_version: "encryptionkey@0"
I20260807 02:56:14.060122 9254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:14.076879 9254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:14.077732 9254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:14.078063 9254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:14.098870 9254 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:45951
I20260807 02:56:14.098891 9342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:45951 every 8 connection(s)
I20260807 02:56:14.100389 9254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:14.105020 9345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:14.106137 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 9254
I20260807 02:56:14.106395 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:14.107121 1509 external_mini_cluster.cc:1428] Setting key 95b9e9df872b0f843d70c7a01cf5f6f4
I20260807 02:56:14.120728 9345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a: Bootstrap starting.
I20260807 02:56:14.121371 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:14.109180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44398 (local address 127.1.121.126:45951)
0807 02:56:14.110290 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:14.110300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:14.110340 (+ 40us) server_negotiation.cc:407] Connection header received
0807 02:56:14.112478 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:14.112501 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:14.113074 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:14.113484 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:14.114058 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:14.115393 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:14.116690 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:14.117442 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:14.117676 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:14.118956 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:14.118964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:14.118978 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:14.119502 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:14.119582 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:14.119603 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:14.119741 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:14.119926 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 02:56:14.120231 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":422,"threads_started":1}
I20260807 02:56:14.124747 9345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:14.125240 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:56:14.125984 9345 log.cc:824] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:14.129227 9345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a: No bootstrap required, opened a new log
I20260807 02:56:14.136340 9345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55583e24acd342968c0a9f990002fc7a" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45951 } }
I20260807 02:56:14.136701 9345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:14.136826 9345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55583e24acd342968c0a9f990002fc7a, State: Initialized, Role: FOLLOWER
I20260807 02:56:14.137423 9345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55583e24acd342968c0a9f990002fc7a" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45951 } }
I20260807 02:56:14.137713 9345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:14.137861 9345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:14.138247 9345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:14.141067 9345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55583e24acd342968c0a9f990002fc7a" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45951 } }
I20260807 02:56:14.141633 9345 leader_election.cc:302] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55583e24acd342968c0a9f990002fc7a; no voters:
I20260807 02:56:14.142299 9345 leader_election.cc:288] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:14.142508 9351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:14.144652 9345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:14.145066 9351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [term 1 LEADER]: Becoming Leader. State: Replica: 55583e24acd342968c0a9f990002fc7a, State: Running, Role: LEADER
I20260807 02:56:14.145641 9351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55583e24acd342968c0a9f990002fc7a" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45951 } }
I20260807 02:56:14.149169 9354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55583e24acd342968c0a9f990002fc7a. Latest consensus state: current_term: 1 leader_uuid: "55583e24acd342968c0a9f990002fc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55583e24acd342968c0a9f990002fc7a" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45951 } } }
I20260807 02:56:14.149591 9354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:14.149359 9353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55583e24acd342968c0a9f990002fc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55583e24acd342968c0a9f990002fc7a" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45951 } } }
I20260807 02:56:14.150838 9353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:14.152978 9368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:14.157780 9368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:14.167945 9368 catalog_manager.cc:1347] Generated new cluster ID: 20034d5b9ba44a0385112d6fccba5e93
I20260807 02:56:14.168102 9368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:14.183369 9368 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:14.185346 9368 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:14.194303 9368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55583e24acd342968c0a9f990002fc7a: Generated new TSK 0
I20260807 02:56:14.195377 9368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:56:14.223953 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:14.215683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.1.121.126:45951)
0807 02:56:14.215903 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:14.215910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:14.216265 (+ 355us) server_negotiation.cc:407] Connection header received
0807 02:56:14.216513 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:14.216517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:14.216587 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:14.216739 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0807 02:56:14.217316 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:14.218070 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:14.220034 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:14.220470 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:14.220568 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:14.221242 (+ 674us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 02:56:14.223275 (+ 2033us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 02:56:14.223630 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":88}
W20260807 02:56:14.224105 9349 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260807 02:56:14.224145 9390 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:14.215477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45951 (local address 127.0.0.1:44410)
0807 02:56:14.216147 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:14.216168 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:14.216332 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:14.216801 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:14.216807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:14.216821 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0807 02:56:14.217158 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:14.217171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:14.218221 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:14.218227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:14.219887 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:14.219903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:14.220678 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:14.220688 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:14.220915 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:14.220964 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 02:56:14.223484 (+ 2520us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 02:56:14.223734 (+ 250us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.121.126:45951: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":335,"threads_started":1}
I20260807 02:56:14.225137 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 9254
I20260807 02:56:14.226426 9337 generic_service.cc:196] Checking for leaks (request via RPC)
E20260807 02:56:14.237811 9391 webserver.cc:533] Webserver: sslize failed:
E20260807 02:56:14.237910 9391 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 02:56:14.319547 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 9254
2026-08-07T02:56:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (774 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T02:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:14Z Disabled control of system clock
I20260807 02:56:14.422817 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:42219
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:41633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:42219
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/testchainca.cert with env {}
W20260807 02:56:14.768582 9399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:14.768992 9399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:14.769093 9399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:14.776618 9399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:14.776743 9399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:14.776782 9399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:14.776883 9399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:14.776960 9399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 02:56:14.777024 9399 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 02:56:14.777058 9399 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 02:56:14.787842 9399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:42219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:14.789307 9399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:14.790622 9399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:14.803148 9406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:14.803176 9414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:14.803412 9408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:14.804880 9399 server_base.cc:1034] running on GCE node
I20260807 02:56:14.805724 9399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:14.807228 9399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:14.808444 9399 hybrid_clock.cc:648] HybridClock initialized: now 1786071374808385 us; error 60 us; skew 500 ppm
I20260807 02:56:14.808933 9399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:56:14.811319 9399 webserver.cc:460] Webserver started at http://127.1.121.126:42913/ using document root <none> and password file <none>
I20260807 02:56:14.812077 9399 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:14.812206 9399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:14.812547 9399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:14.815657 9399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "f3860b6443804d6c81b9cbcdd593879f"
format_stamp: "Formatted at 2026-08-07 02:56:14 on dist-test-slave-3h5h"
server_key: "55cd9839b472ad5df59d18d216df8fa0"
server_key_iv: "3e53099fd8cb80757ed555a2ba1eafd8"
server_key_version: "encryptionkey@0"
I20260807 02:56:14.816497 9399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "f3860b6443804d6c81b9cbcdd593879f"
format_stamp: "Formatted at 2026-08-07 02:56:14 on dist-test-slave-3h5h"
server_key: "55cd9839b472ad5df59d18d216df8fa0"
server_key_iv: "3e53099fd8cb80757ed555a2ba1eafd8"
server_key_version: "encryptionkey@0"
I20260807 02:56:14.821791 9399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260807 02:56:14.825414 9422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:14.826876 9399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:56:14.827072 9399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "f3860b6443804d6c81b9cbcdd593879f"
format_stamp: "Formatted at 2026-08-07 02:56:14 on dist-test-slave-3h5h"
server_key: "55cd9839b472ad5df59d18d216df8fa0"
server_key_iv: "3e53099fd8cb80757ed555a2ba1eafd8"
server_key_version: "encryptionkey@0"
I20260807 02:56:14.827383 9399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:14.847427 9399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:14.848340 9399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:14.848659 9399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:14.868891 9399 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:42219
I20260807 02:56:14.868944 9489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:42219 every 8 connection(s)
I20260807 02:56:14.870399 9399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:14.874413 9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:14.878068 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 9399
I20260807 02:56:14.878297 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:14.878970 1509 external_mini_cluster.cc:1428] Setting key 7fe7b2139e588777dfb732f83cf5a58a
I20260807 02:56:14.886220 9490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f: Bootstrap starting.
I20260807 02:56:14.889608 9490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:14.890810 9490 log.cc:824] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:14.893960 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:14.880999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.1.121.126:42219)
0807 02:56:14.882137 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:14.882150 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:14.882196 (+ 46us) server_negotiation.cc:407] Connection header received
0807 02:56:14.884597 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:14.884664 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:14.885254 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:14.885678 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:14.886278 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:14.887661 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:14.889062 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:14.889828 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:14.890057 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:14.891288 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:14.891318 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:14.891341 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:56:14.892157 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:14.892264 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:56:14.892293 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:14.892432 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:14.892679 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 02:56:14.893037 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":424,"threads_started":1}
I20260807 02:56:14.894253 9490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f: No bootstrap required, opened a new log
I20260807 02:56:14.897992 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:56:14.902674 9490 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3860b6443804d6c81b9cbcdd593879f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42219 } }
I20260807 02:56:14.903043 9490 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:14.903172 9490 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3860b6443804d6c81b9cbcdd593879f, State: Initialized, Role: FOLLOWER
I20260807 02:56:14.903724 9490 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3860b6443804d6c81b9cbcdd593879f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42219 } }
I20260807 02:56:14.903959 9490 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:14.904127 9490 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:14.904301 9490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:14.906800 9490 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3860b6443804d6c81b9cbcdd593879f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42219 } }
I20260807 02:56:14.907332 9490 leader_election.cc:302] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3860b6443804d6c81b9cbcdd593879f; no voters:
I20260807 02:56:14.907922 9490 leader_election.cc:288] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:14.908097 9497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:14.908883 9497 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [term 1 LEADER]: Becoming Leader. State: Replica: f3860b6443804d6c81b9cbcdd593879f, State: Running, Role: LEADER
I20260807 02:56:14.909420 9497 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3860b6443804d6c81b9cbcdd593879f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42219 } }
I20260807 02:56:14.910231 9490 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:14.912775 9500 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3860b6443804d6c81b9cbcdd593879f. Latest consensus state: current_term: 1 leader_uuid: "f3860b6443804d6c81b9cbcdd593879f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3860b6443804d6c81b9cbcdd593879f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42219 } } }
I20260807 02:56:14.913314 9500 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:14.913304 9499 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3860b6443804d6c81b9cbcdd593879f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3860b6443804d6c81b9cbcdd593879f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42219 } } }
I20260807 02:56:14.913712 9499 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:14.915570 9507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:14.921075 9507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:14.930222 9507 catalog_manager.cc:1347] Generated new cluster ID: d8b4ba23f83f4f26b3d8a68f92b33b01
I20260807 02:56:14.930357 9507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:14.960657 9507 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:14.962117 9507 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:14.972427 9507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3860b6443804d6c81b9cbcdd593879f: Generated new TSK 0
I20260807 02:56:14.973553 9507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:56:16.665022 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:15.003003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.1.121.126:42219)
0807 02:56:15.003209 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:15.003218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:15.003699 (+ 481us) server_negotiation.cc:407] Connection header received
0807 02:56:15.003936 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:15.003943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:15.004056 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:15.004235 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0807 02:56:15.004699 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:15.005891 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:15.007363 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:15.008071 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:15.008197 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:15.008712 (+ 515us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 02:56:16.664371 (+1655659us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 02:56:16.664708 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50968: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 02:56:16.665128 9493 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50968: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 02:56:16.665477 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:15.002933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42219 (local address 127.0.0.1:50968)
0807 02:56:15.003595 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:15.003612 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:15.003745 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:15.004257 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:15.004260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:15.004271 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0807 02:56:15.004543 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:15.004551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:15.006050 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:15.006054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:15.007217 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:15.007230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:15.008247 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:15.008251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:15.008373 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:15.008408 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 02:56:16.664710 (+1656302us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 02:56:16.665043 (+ 333us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.121.126:42219: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":371,"threads_started":1}
I20260807 02:56:16.666786 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 9399
I20260807 02:56:16.667984 9485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:16.772680 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 9399
2026-08-07T02:56:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (2434 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (452 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (454 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:56:17 dist-test-slave-3h5h krb5kdc[9554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:56:17 dist-test-slave-3h5h krb5kdc[9554](info): set up 2 sockets
Aug 07 02:56:17 dist-test-slave-3h5h krb5kdc[9554](info): commencing operation
krb5kdc: starting...
W20260807 02:56:19.039721 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.312s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:56:19 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:56:19.309476 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:38267
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:38267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:19.658106 9575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:19.658551 9575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:19.658751 9575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:19.667237 9575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:19.667349 9575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:19.667465 9575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:19.667578 9575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:19.667641 9575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:19.677925 9575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:38267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:38267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:19.679312 9575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:19.680735 9575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:19.693106 9586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:19.693073 9583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:19.693188 9582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:19.698446 9575 server_base.cc:1034] running on GCE node
I20260807 02:56:19.699282 9575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:19.700727 9575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:19.701958 9575 hybrid_clock.cc:648] HybridClock initialized: now 1786071379701902 us; error 29 us; skew 500 ppm
I20260807 02:56:19.702427 9575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:19 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071379, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:19.710610 9575 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:56:19.712273 9575 webserver.cc:460] Webserver started at http://127.1.121.126:44005/ using document root <none> and password file <none>
I20260807 02:56:19.713093 9575 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:19.713241 9575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:19.713605 9575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:19.716920 9575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "ac5acbcad47c49439d26ae0a8df06281"
format_stamp: "Formatted at 2026-08-07 02:56:19 on dist-test-slave-3h5h"
server_key: "2a72c68eade7e58e449f614a265da393"
server_key_iv: "34d312f12a9ce19f079aa6dde5f4013f"
server_key_version: "encryptionkey@0"
I20260807 02:56:19.717759 9575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "ac5acbcad47c49439d26ae0a8df06281"
format_stamp: "Formatted at 2026-08-07 02:56:19 on dist-test-slave-3h5h"
server_key: "2a72c68eade7e58e449f614a265da393"
server_key_iv: "34d312f12a9ce19f079aa6dde5f4013f"
server_key_version: "encryptionkey@0"
I20260807 02:56:19.723233 9575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 02:56:19.726846 9595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:19.728188 9575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:56:19.728370 9575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "ac5acbcad47c49439d26ae0a8df06281"
format_stamp: "Formatted at 2026-08-07 02:56:19 on dist-test-slave-3h5h"
server_key: "2a72c68eade7e58e449f614a265da393"
server_key_iv: "34d312f12a9ce19f079aa6dde5f4013f"
server_key_version: "encryptionkey@0"
I20260807 02:56:19.728566 9575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:19.747193 9575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:19.751407 9575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:19.751710 9575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:19.770854 9575 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:38267
I20260807 02:56:19.770891 9661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:38267 every 8 connection(s)
I20260807 02:56:19.772264 9575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:19.775475 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 9575
I20260807 02:56:19.775717 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:19.776237 9662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:19.776450 1509 external_mini_cluster.cc:1428] Setting key 0058eca487cdcfa46eb54b600c7789b9
I20260807 02:56:19.789304 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281: Bootstrap starting.
Aug 07 02:56:19 dist-test-slave-3h5h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:19.792945 9662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:19.794221 9662 log.cc:824] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:19.797210 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281: No bootstrap required, opened a new log
I20260807 02:56:19.799558 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:19.778597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.1.121.126:38267)
0807 02:56:19.779441 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:19.779452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:19.779494 (+ 42us) server_negotiation.cc:407] Connection header received
0807 02:56:19.781294 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:19.781318 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:19.781710 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:19.782125 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:19.783627 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:19.784978 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:19.786261 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:19.787005 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:19.787232 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:19.791530 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:19.791538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:19.791552 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:19.791595 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:19.794946 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:19.795670 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:19.795676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:19.795683 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:19.795843 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:19.796248 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:19.796254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:19.796257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:19.796760 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:19.797092 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:19.797677 (+ 585us) server_negotiation.cc:299] Negotiation successful
0807 02:56:19.798208 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":363,"threads_started":1}
I20260807 02:56:19.806093 9662 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5acbcad47c49439d26ae0a8df06281" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38267 } }
I20260807 02:56:19.806432 9662 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:19.806505 9662 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5acbcad47c49439d26ae0a8df06281, State: Initialized, Role: FOLLOWER
I20260807 02:56:19.807003 9662 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5acbcad47c49439d26ae0a8df06281" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38267 } }
I20260807 02:56:19.807212 9662 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:19.807338 9662 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:19.807467 9662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:19.809974 9662 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5acbcad47c49439d26ae0a8df06281" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38267 } }
I20260807 02:56:19.810433 9662 leader_election.cc:302] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac5acbcad47c49439d26ae0a8df06281; no voters:
I20260807 02:56:19.810977 9662 leader_election.cc:288] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:19.811157 9671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:19.811944 9671 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [term 1 LEADER]: Becoming Leader. State: Replica: ac5acbcad47c49439d26ae0a8df06281, State: Running, Role: LEADER
I20260807 02:56:19.812537 9671 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5acbcad47c49439d26ae0a8df06281" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38267 } }
I20260807 02:56:19.813376 9662 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:19.815588 9673 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac5acbcad47c49439d26ae0a8df06281. Latest consensus state: current_term: 1 leader_uuid: "ac5acbcad47c49439d26ae0a8df06281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5acbcad47c49439d26ae0a8df06281" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38267 } } }
I20260807 02:56:19.816103 9673 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:19.818393 9678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:19.818367 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac5acbcad47c49439d26ae0a8df06281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5acbcad47c49439d26ae0a8df06281" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38267 } } }
I20260807 02:56:19.818790 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:19.824149 9678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:19.833431 9678 catalog_manager.cc:1347] Generated new cluster ID: de91758a03974c5984657a157162e6e9
I20260807 02:56:19.833565 9678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:19.847651 9678 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:19.849901 9678 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:19.861626 9678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac5acbcad47c49439d26ae0a8df06281: Generated new TSK 0
I20260807 02:56:19.862653 9678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:56:19.929085 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38267
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:20.280383 9699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:20.280678 9699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:20.280889 9699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:20.288472 9699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:20.288623 9699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:20.288820 9699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:56:20.300344 9699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:20.301780 9699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:20.303123 9699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:20.317123 9705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:20.317168 9707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:20.317205 9712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:20.318339 9699 server_base.cc:1034] running on GCE node
I20260807 02:56:20.319027 9699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:20.320240 9699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:20.321518 9699 hybrid_clock.cc:648] HybridClock initialized: now 1786071380321426 us; error 66 us; skew 500 ppm
I20260807 02:56:20.321908 9699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:20 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071380, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:20.325835 9699 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:56:20.327596 9699 webserver.cc:460] Webserver started at http://127.1.121.65:42847/ using document root <none> and password file <none>
I20260807 02:56:20.328341 9699 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:20.328469 9699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:20.328786 9699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:20.332254 9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "f8d2d212c8a7417fa54e3b894a9eed9e"
format_stamp: "Formatted at 2026-08-07 02:56:20 on dist-test-slave-3h5h"
server_key: "e0775345d6510c2df0e254c70a80da50"
server_key_iv: "abb48725843e23bde6c77ed9d0ddfc09"
server_key_version: "encryptionkey@0"
I20260807 02:56:20.333344 9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "f8d2d212c8a7417fa54e3b894a9eed9e"
format_stamp: "Formatted at 2026-08-07 02:56:20 on dist-test-slave-3h5h"
server_key: "e0775345d6510c2df0e254c70a80da50"
server_key_iv: "abb48725843e23bde6c77ed9d0ddfc09"
server_key_version: "encryptionkey@0"
I20260807 02:56:20.339480 9699 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260807 02:56:20.343514 9721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:20.345070 9699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:20.345252 9699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "f8d2d212c8a7417fa54e3b894a9eed9e"
format_stamp: "Formatted at 2026-08-07 02:56:20 on dist-test-slave-3h5h"
server_key: "e0775345d6510c2df0e254c70a80da50"
server_key_iv: "abb48725843e23bde6c77ed9d0ddfc09"
server_key_version: "encryptionkey@0"
I20260807 02:56:20.345465 9699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:20.379792 9699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:20.384644 9699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:20.385020 9699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:20.386125 9699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:20.387660 9699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:20.387785 9699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:20.387964 9699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:20.388056 9699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:20.425429 9699 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:42755
I20260807 02:56:20.425462 9878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:42755 every 8 connection(s)
I20260807 02:56:20.427301 9699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:56:20.436739 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 9699
I20260807 02:56:20.436995 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:56:20.437673 1509 external_mini_cluster.cc:1428] Setting key ca5d796ffc7b2607dac87eed20aaf07a
Aug 07 02:56:20 dist-test-slave-3h5h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071380, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:20.448982 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:20.430326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:49267 (local address 127.1.121.126:38267)
0807 02:56:20.430776 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:20.430781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:20.431951 (+ 1170us) server_negotiation.cc:407] Connection header received
0807 02:56:20.433987 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:20.433992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:20.434064 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:20.434176 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:20.435673 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:20.436577 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:20.438640 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:20.439086 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:20.439227 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:20.442931 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:20.442936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:20.442939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:20.442979 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:20.445828 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:20.446590 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:20.446600 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:20.446602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:20.446718 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:20.447166 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:20.447172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:20.447177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:20.447444 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:20.447604 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:20.448327 (+ 723us) server_negotiation.cc:299] Negotiation successful
0807 02:56:20.448561 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260807 02:56:20.450070 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:20.430936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:38267 (local address 127.1.121.65:49267)
0807 02:56:20.431775 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:20.431839 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:20.433711 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:20.434388 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:20.434395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:20.434945 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:20.435504 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:20.435520 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:20.436775 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:20.436780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:20.438475 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:20.438494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:20.439342 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:20.439351 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:20.439619 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:20.440313 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:20.440341 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:20.442701 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:20.445981 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:20.445992 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:20.446010 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:20.446445 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:20.446834 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:20.446839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:20.446843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:20.447041 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:20.447582 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:20.447590 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:20.447939 (+ 349us) client_negotiation.cc:769] Sending connection context
0807 02:56:20.448214 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 02:56:20.448810 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":283,"threads_started":1}
I20260807 02:56:20.451730 9879 heartbeater.cc:347] Connected to a master server at 127.1.121.126:38267
I20260807 02:56:20.452060 9879 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:20.453009 9879 heartbeater.cc:511] Master 127.1.121.126:38267 requested a full tablet report, sending...
I20260807 02:56:20.455685 9617 ts_manager.cc:194] Registered new tserver with Master: f8d2d212c8a7417fa54e3b894a9eed9e (127.1.121.65:42755)
I20260807 02:56:20.457975 9617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:49267
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:56:20.500440 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38267
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:20.850651 9885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:20.850992 9885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:20.851213 9885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:20.858736 9885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:20.858896 9885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:20.859069 9885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:56:20.869052 9885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:20.870424 9885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:20.871822 9885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:20.885418 9892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:20.885440 9891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:20.885360 9896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:20.886238 9885 server_base.cc:1034] running on GCE node
I20260807 02:56:20.886818 9885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:20.887740 9885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:20.888958 9885 hybrid_clock.cc:648] HybridClock initialized: now 1786071380888914 us; error 37 us; skew 500 ppm
I20260807 02:56:20.889333 9885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:20 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071380, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:20.893015 9885 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:56:20.894579 9885 webserver.cc:460] Webserver started at http://127.1.121.66:43773/ using document root <none> and password file <none>
I20260807 02:56:20.895272 9885 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:20.895376 9885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:20.895715 9885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:20.898743 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "b294a6cd825246a59235f244e6bf8072"
format_stamp: "Formatted at 2026-08-07 02:56:20 on dist-test-slave-3h5h"
server_key: "3bafee068ca4b4b87dce2d42c13946d4"
server_key_iv: "bd6fd46d28e0b03c589c5dd38aaa1c1c"
server_key_version: "encryptionkey@0"
I20260807 02:56:20.899601 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "b294a6cd825246a59235f244e6bf8072"
format_stamp: "Formatted at 2026-08-07 02:56:20 on dist-test-slave-3h5h"
server_key: "3bafee068ca4b4b87dce2d42c13946d4"
server_key_iv: "bd6fd46d28e0b03c589c5dd38aaa1c1c"
server_key_version: "encryptionkey@0"
I20260807 02:56:20.904644 9885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 02:56:20.908016 9907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:20.909425 9885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:20.909605 9885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "b294a6cd825246a59235f244e6bf8072"
format_stamp: "Formatted at 2026-08-07 02:56:20 on dist-test-slave-3h5h"
server_key: "3bafee068ca4b4b87dce2d42c13946d4"
server_key_iv: "bd6fd46d28e0b03c589c5dd38aaa1c1c"
server_key_version: "encryptionkey@0"
I20260807 02:56:20.909808 9885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:20.929889 9885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:20.935308 9885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:20.935755 9885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:20.936993 9885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:20.938500 9885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:20.938576 9885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:20.938736 9885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:20.938788 9885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:20.974128 9885 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:44775
I20260807 02:56:20.974221 10057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:44775 every 8 connection(s)
I20260807 02:56:20.975455 9885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:56:20.977942 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 9885
I20260807 02:56:20.978150 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:56:20.978724 1509 external_mini_cluster.cc:1428] Setting key 1185c42ca68e9e9257e40768eb136cfe
Aug 07 02:56:20 dist-test-slave-3h5h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071380, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:20.999119 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:20.978205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:47435 (local address 127.1.121.126:38267)
0807 02:56:20.978713 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:20.978720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:20.979883 (+ 1163us) server_negotiation.cc:407] Connection header received
0807 02:56:20.982384 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:20.982388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:20.982453 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:20.982556 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:20.984768 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:20.985606 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:20.988120 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:20.988590 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:20.988694 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:20.993108 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:20.993122 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:20.993127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:20.993163 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:20.995871 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:20.996665 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:20.996671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:20.996681 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:20.996806 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:20.997312 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:20.997317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:20.997320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:20.997604 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:20.997764 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:20.998480 (+ 716us) server_negotiation.cc:299] Negotiation successful
0807 02:56:20.998724 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":233,"threads_started":1}
I20260807 02:56:21.000303 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:20.978799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:38267 (local address 127.1.121.66:47435)
0807 02:56:20.979697 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:20.979754 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:20.982073 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:20.982849 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:20.982864 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:20.983708 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:20.984548 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:20.984584 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:20.985769 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:20.985777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:20.987945 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:20.987963 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:20.988868 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:20.988879 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:20.989220 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:20.990025 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:20.990068 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:20.992799 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:20.996049 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:20.996059 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:20.996079 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:20.996522 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:20.996961 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:20.996966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:20.996970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:20.997181 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:20.997738 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:20.997747 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:20.998111 (+ 364us) client_negotiation.cc:769] Sending connection context
0807 02:56:20.998402 (+ 291us) client_negotiation.cc:240] Negotiation successful
0807 02:56:20.998965 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":335,"threads_started":1}
I20260807 02:56:21.001914 10061 heartbeater.cc:347] Connected to a master server at 127.1.121.126:38267
I20260807 02:56:21.002317 10061 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:21.003373 10061 heartbeater.cc:511] Master 127.1.121.126:38267 requested a full tablet report, sending...
I20260807 02:56:21.005272 9617 ts_manager.cc:194] Registered new tserver with Master: b294a6cd825246a59235f244e6bf8072 (127.1.121.66:44775)
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
I20260807 02:56:21.006495 9617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:47435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:56:21.042522 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38267
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:21.397323 10067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:21.397614 10067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:21.397724 10067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:21.405640 10067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:21.405759 10067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:21.405872 10067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:56:21.416153 10067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:21.417490 10067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:21.418813 10067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:21.433377 10074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:21.433243 10078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:21.433279 10075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:21.434480 10067 server_base.cc:1034] running on GCE node
I20260807 02:56:21.435153 10067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:21.436149 10067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:21.437381 10067 hybrid_clock.cc:648] HybridClock initialized: now 1786071381437337 us; error 30 us; skew 500 ppm
I20260807 02:56:21.437713 10067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:21.440819 10067 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:56:21.442394 10067 webserver.cc:460] Webserver started at http://127.1.121.67:37891/ using document root <none> and password file <none>
I20260807 02:56:21.443123 10067 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:21.443274 10067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:21.443595 10067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:21.446595 10067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "eb335c5b927d46c3831b588b6d6ae0ab"
format_stamp: "Formatted at 2026-08-07 02:56:21 on dist-test-slave-3h5h"
server_key: "1c6abaf418c7f061c6d980b91eb68785"
server_key_iv: "89f53855f5dba91bf17ce13defc39abf"
server_key_version: "encryptionkey@0"
I20260807 02:56:21.447393 10067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "eb335c5b927d46c3831b588b6d6ae0ab"
format_stamp: "Formatted at 2026-08-07 02:56:21 on dist-test-slave-3h5h"
server_key: "1c6abaf418c7f061c6d980b91eb68785"
server_key_iv: "89f53855f5dba91bf17ce13defc39abf"
server_key_version: "encryptionkey@0"
I20260807 02:56:21.452762 10067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 02:56:21.456247 10088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:21.457617 10067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:21.457803 10067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "eb335c5b927d46c3831b588b6d6ae0ab"
format_stamp: "Formatted at 2026-08-07 02:56:21 on dist-test-slave-3h5h"
server_key: "1c6abaf418c7f061c6d980b91eb68785"
server_key_iv: "89f53855f5dba91bf17ce13defc39abf"
server_key_version: "encryptionkey@0"
I20260807 02:56:21.458020 10067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:21.461920 9879 heartbeater.cc:503] Master 127.1.121.126:38267 was elected leader, sending a full tablet report...
I20260807 02:56:21.482200 10067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:21.486418 10067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:21.486725 10067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:21.487784 10067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:21.489244 10067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:21.489352 10067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:21.489506 10067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:21.489580 10067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:21.524761 10067 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:33115
I20260807 02:56:21.524807 10248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:33115 every 8 connection(s)
I20260807 02:56:21.526116 10067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:56:21.529855 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 10067
I20260807 02:56:21.530053 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:56:21.530669 1509 external_mini_cluster.cc:1428] Setting key 364090de32edda4becf3aa93349cadaf
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:21.547142 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:21.528909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:38785 (local address 127.1.121.126:38267)
0807 02:56:21.529343 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.529347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.530596 (+ 1249us) server_negotiation.cc:407] Connection header received
0807 02:56:21.532887 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.532892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.532999 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.533120 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.534840 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.535617 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.537173 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.537555 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.537650 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:21.540937 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:21.540942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:21.540945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:21.540977 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:21.543746 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.544535 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.544538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.544541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.544662 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.545202 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.545212 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.545218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.545633 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:21.545777 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:21.546630 (+ 853us) server_negotiation.cc:299] Negotiation successful
0807 02:56:21.546833 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260807 02:56:21.548110 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:21.529522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:38267 (local address 127.1.121.67:38785)
0807 02:56:21.530400 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:21.530460 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:21.532550 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:21.533366 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:21.533373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:21.533976 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:21.534678 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:21.534696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.535753 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.535757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:21.537043 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:21.537056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.537748 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.537753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:21.537938 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:21.538568 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:21.538589 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:21.540708 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:21.543948 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:21.543955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:21.543966 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:21.544415 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:21.544796 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:21.544800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:21.544802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:21.545075 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:21.545771 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:21.545781 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:21.546188 (+ 407us) client_negotiation.cc:769] Sending connection context
0807 02:56:21.546481 (+ 293us) client_negotiation.cc:240] Negotiation successful
0807 02:56:21.546991 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":353,"threads_started":1}
I20260807 02:56:21.549532 10249 heartbeater.cc:347] Connected to a master server at 127.1.121.126:38267
I20260807 02:56:21.549794 10249 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:21.550518 10249 heartbeater.cc:511] Master 127.1.121.126:38267 requested a full tablet report, sending...
I20260807 02:56:21.551882 9617 ts_manager.cc:194] Registered new tserver with Master: eb335c5b927d46c3831b588b6d6ae0ab (127.1.121.67:33115)
I20260807 02:56:21.552698 9617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:38785
I20260807 02:56:21.558781 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/test-admin.keytab.
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:21.592199 1509 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 02:56:21.597743 10258 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:21.595004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.1.121.65:42755)
0807 02:56:21.595700 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.595711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.595745 (+ 34us) server_negotiation.cc:407] Connection header received
0807 02:56:21.595927 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.595935 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.596168 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.596341 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.597290 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39400: BlockingRecv error: recv got EOF from 127.0.0.1:39400 (error 108)
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":320,"threads_started":1}
W20260807 02:56:21.598129 1509 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5cc' of type FILE
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:21.600063 1509 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:21.601871 1509 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:56:21.618350 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:21.603712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.1.121.65:42755)
0807 02:56:21.603975 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.603981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.604290 (+ 309us) server_negotiation.cc:407] Connection header received
0807 02:56:21.604455 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.604459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.604519 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.604606 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.605834 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.606801 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.608125 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.608681 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.608785 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:21.612361 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:21.612373 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:21.612389 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:21.612421 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:21.615538 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.616263 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.616270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.616272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.616389 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.616785 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.616788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.616790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.617239 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:21.617459 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:21.617767 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 02:56:21.618049 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 02:56:21.624334 10258 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:21.622643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.1.121.65:42755)
0807 02:56:21.622870 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.622876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.623122 (+ 246us) server_negotiation.cc:407] Connection header received
0807 02:56:21.623255 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.623258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.623313 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.623388 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.624028 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39414: BlockingRecv error: recv got EOF from 127.0.0.1:39414 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 02:56:21.624622 1509 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5cc' of type FILE
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:21.626550 1509 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:21.628104 1509 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:56:21.642709 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:21.629411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.1.121.65:42755)
0807 02:56:21.629626 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.629633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.629833 (+ 200us) server_negotiation.cc:407] Connection header received
0807 02:56:21.629954 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.629957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.630012 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.630116 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.630962 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.631692 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.633006 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.633426 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.633521 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:21.637069 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:21.637076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:21.637080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:21.637117 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:21.640034 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.640628 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.640643 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.640646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.640791 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.641322 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.641326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.641328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.641656 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:21.641882 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:21.642152 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 02:56:21.642422 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260807 02:56:21.647115 10258 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:21.645503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.1.121.65:42755)
0807 02:56:21.645688 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.645692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.645763 (+ 71us) server_negotiation.cc:407] Connection header received
0807 02:56:21.645887 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.645890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.645956 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.646031 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.646851 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39438: BlockingRecv error: recv got EOF from 127.0.0.1:39438 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 02:56:21.647250 1509 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786071323612974-1509-0/krb5kdc/krb5cc' of type FILE
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:21.648810 1509 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:21.650254 1509 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:56:21.665103 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:21.651578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.1.121.65:42755)
0807 02:56:21.651816 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.651824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.652108 (+ 284us) server_negotiation.cc:407] Connection header received
0807 02:56:21.652311 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.652316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.652386 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.652488 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.653532 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.654268 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.655505 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.655940 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.656074 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:21.659515 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:21.659521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:21.659528 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:21.659565 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:21.662363 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.663032 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.663039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.663042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.663190 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.663572 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.663577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.663580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.663850 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:21.664209 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:21.664445 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 02:56:21.664659 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 02:56:21.667698 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 9699
I20260807 02:56:21.679072 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:21.668452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.1.121.65:42755)
0807 02:56:21.668613 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.668617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.668982 (+ 365us) server_negotiation.cc:407] Connection header received
0807 02:56:21.669116 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.669119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.669174 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.669251 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.670108 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.670769 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.671962 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.672358 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.672448 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:21.674465 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:21.674470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:21.674473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:21.674501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:21.676900 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.677507 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.677510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.677512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.677643 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.677972 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.677975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.677977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.678221 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:21.678422 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:21.678677 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 02:56:21.678832 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 02:56:21.679669 9872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:21.763872 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 9699
I20260807 02:56:21.794767 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 9885
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:56:21.810755 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:21.795866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47878 (local address 127.1.121.66:44775)
0807 02:56:21.796393 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.796400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.796426 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:56:21.796491 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.796497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.796717 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.796929 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.798008 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.799028 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.800325 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.800949 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.801092 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:21.805051 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:21.805063 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:21.805080 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:21.805114 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:21.807888 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.808605 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.808613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.808615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.808738 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.809153 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.809158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.809161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.809506 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:21.809849 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:21.810171 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 02:56:21.810438 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":264,"threads_started":1}
I20260807 02:56:21.812052 10052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:21.896708 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 9885
I20260807 02:56:21.922652 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 10067
Aug 07 02:56:21 dist-test-slave-3h5h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:56:21.939610 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:21.923782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.1.121.67:33115)
0807 02:56:21.924411 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:21.924418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:21.924449 (+ 31us) server_negotiation.cc:407] Connection header received
0807 02:56:21.924511 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:21.924517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:21.924732 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:21.924928 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:21.926193 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.927320 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.928626 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:21.929387 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:21.929488 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:21.933684 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:21.933707 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:21.933722 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:21.933761 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:21.936672 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.937413 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.937421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.937424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.937561 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:21.937982 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:21.937987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:21.937989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:21.938345 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:21.938669 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:21.938991 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 02:56:21.939292 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":316,"threads_started":1}
I20260807 02:56:21.941147 10242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:22.026270 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 10067
I20260807 02:56:22.050895 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 9575
I20260807 02:56:22.052170 9657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:22.194801 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 9575
2026-08-07T02:56:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4570 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:56:22 dist-test-slave-3h5h krb5kdc[10290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:56:22 dist-test-slave-3h5h krb5kdc[10290](info): set up 2 sockets
Aug 07 02:56:22 dist-test-slave-3h5h krb5kdc[10290](info): commencing operation
krb5kdc: starting...
W20260807 02:56:24.180347 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.880s user 0.002s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:56:24 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.126.keytab.
Entry for principal oryx/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.126.keytab.
I20260807 02:56:24.464655 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41229
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.126.keytab
--principal=oryx/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:24.815244 10311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:24.815608 10311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:24.815794 10311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:24.823081 10311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:24.823248 10311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:24.823339 10311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:24.823433 10311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:24.823509 10311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:24.833842 10311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.126.keytab
--principal=oryx/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:24.835244 10311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:24.836711 10311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:24.849045 10316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:24.849134 10319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:24.849294 10321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:24.850380 10311 server_base.cc:1034] running on GCE node
I20260807 02:56:24.851239 10311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:24.852736 10311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:24.853951 10311 hybrid_clock.cc:648] HybridClock initialized: now 1786071384853882 us; error 59 us; skew 500 ppm
I20260807 02:56:24.854401 10311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:24 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:24.862296 10311 init.cc:377] Logged in from keytab as oryx/127.1.121.126@KRBTEST.COM (short username oryx)
I20260807 02:56:24.864097 10311 webserver.cc:460] Webserver started at http://127.1.121.126:44663/ using document root <none> and password file <none>
I20260807 02:56:24.864954 10311 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:24.865132 10311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:24.865497 10311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:24.868794 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9"
format_stamp: "Formatted at 2026-08-07 02:56:24 on dist-test-slave-3h5h"
server_key: "67c1adc3805ae63597eaf885c3501362"
server_key_iv: "39eb6ecad34b8c6cd61fb80396495085"
server_key_version: "encryptionkey@0"
I20260807 02:56:24.869828 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9"
format_stamp: "Formatted at 2026-08-07 02:56:24 on dist-test-slave-3h5h"
server_key: "67c1adc3805ae63597eaf885c3501362"
server_key_iv: "39eb6ecad34b8c6cd61fb80396495085"
server_key_version: "encryptionkey@0"
I20260807 02:56:24.875663 10311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 02:56:24.879619 10331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:24.881239 10311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 02:56:24.881433 10311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9"
format_stamp: "Formatted at 2026-08-07 02:56:24 on dist-test-slave-3h5h"
server_key: "67c1adc3805ae63597eaf885c3501362"
server_key_iv: "39eb6ecad34b8c6cd61fb80396495085"
server_key_version: "encryptionkey@0"
I20260807 02:56:24.881646 10311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:24.895021 10311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:24.900030 10311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:24.900364 10311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:24.921859 10311 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:41229
I20260807 02:56:24.921898 10406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:41229 every 8 connection(s)
I20260807 02:56:24.923456 10311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:24.927278 10408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:24.931680 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 10311
I20260807 02:56:24.931905 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:24.932775 1509 external_mini_cluster.cc:1428] Setting key 4deb87e9aa70cc1fbdc0d2afe97a3948
I20260807 02:56:24.940330 10408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: Bootstrap starting.
I20260807 02:56:24.943583 10408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:24.944602 10408 log.cc:824] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:24.947883 10408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: No bootstrap required, opened a new log
Aug 07 02:56:24 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:24.956542 10408 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } }
I20260807 02:56:24.957008 10408 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:24.957139 10408 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcbe99e52f743b7ab27a4fba0ccfec9, State: Initialized, Role: FOLLOWER
I20260807 02:56:24.957864 10408 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } }
I20260807 02:56:24.958118 10408 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:24.958266 10408 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:24.958469 10408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:24.960126 10411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:24.935250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.1.121.126:41229)
0807 02:56:24.936245 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:24.936262 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:24.936325 (+ 63us) server_negotiation.cc:407] Connection header received
0807 02:56:24.939183 (+ 2858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:24.939210 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:24.939650 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:24.940137 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:24.941902 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:24.943334 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:24.944819 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:24.946118 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:24.946325 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:24.951512 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:24.951525 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:24.951541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:24.951587 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:24.955239 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:24.956054 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:24.956069 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:24.956077 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:24.956270 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:24.956730 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:24.956741 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:24.956743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:24.957540 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:24.957918 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:24.958403 (+ 485us) server_negotiation.cc:299] Negotiation successful
0807 02:56:24.958905 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":404,"threads_started":1}
I20260807 02:56:24.961539 10408 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } }
I20260807 02:56:24.962101 10408 leader_election.cc:302] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfcbe99e52f743b7ab27a4fba0ccfec9; no voters:
I20260807 02:56:24.962777 10408 leader_election.cc:288] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:24.962975 10414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:24.964771 10414 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 1 LEADER]: Becoming Leader. State: Replica: dfcbe99e52f743b7ab27a4fba0ccfec9, State: Running, Role: LEADER
I20260807 02:56:24.964994 10408 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:24.965412 10414 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } }
I20260807 02:56:24.968811 10415 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } }
I20260807 02:56:24.969302 10415 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:24.969501 10417 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcbe99e52f743b7ab27a4fba0ccfec9. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } }
I20260807 02:56:24.969870 10417 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:24.973160 10428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:24.978674 10428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:24.991367 10428 catalog_manager.cc:1347] Generated new cluster ID: b7c877b6dc8a48c1b68079f078964abb
I20260807 02:56:24.991503 10428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:25.013073 10428 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:25.015106 10428 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:25.027613 10428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: Generated new TSK 0
I20260807 02:56:25.028774 10428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab.
Entry for principal oryx/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab.
I20260807 02:56:25.101620 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41229
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab
--principal=oryx/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:25.469583 10446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:25.469866 10446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:25.470059 10446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:25.477808 10446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:25.477926 10446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:25.478142 10446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:56:25.488492 10446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab
--principal=oryx/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:25.489887 10446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:25.491294 10446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:25.505699 10456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:25.505784 10455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:25.505851 10458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:25.506393 10446 server_base.cc:1034] running on GCE node
I20260807 02:56:25.507026 10446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:25.508018 10446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:25.509259 10446 hybrid_clock.cc:648] HybridClock initialized: now 1786071385509201 us; error 58 us; skew 500 ppm
I20260807 02:56:25.509594 10446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:25 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071385, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:25.512916 10446 init.cc:377] Logged in from keytab as oryx/127.1.121.65@KRBTEST.COM (short username oryx)
I20260807 02:56:25.514554 10446 webserver.cc:460] Webserver started at http://127.1.121.65:46675/ using document root <none> and password file <none>
I20260807 02:56:25.515312 10446 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:25.515456 10446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:25.515770 10446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:25.518815 10446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "313d60cfddb645daaf6e649f261168c5"
format_stamp: "Formatted at 2026-08-07 02:56:25 on dist-test-slave-3h5h"
server_key: "bf9cfa7d717243f01bbd31a3b5c24106"
server_key_iv: "6ad14d38a2b17b28a151351982aacdce"
server_key_version: "encryptionkey@0"
I20260807 02:56:25.519680 10446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "313d60cfddb645daaf6e649f261168c5"
format_stamp: "Formatted at 2026-08-07 02:56:25 on dist-test-slave-3h5h"
server_key: "bf9cfa7d717243f01bbd31a3b5c24106"
server_key_iv: "6ad14d38a2b17b28a151351982aacdce"
server_key_version: "encryptionkey@0"
I20260807 02:56:25.524991 10446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 02:56:25.528967 10468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:25.530637 10446 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:56:25.530812 10446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "313d60cfddb645daaf6e649f261168c5"
format_stamp: "Formatted at 2026-08-07 02:56:25 on dist-test-slave-3h5h"
server_key: "bf9cfa7d717243f01bbd31a3b5c24106"
server_key_iv: "6ad14d38a2b17b28a151351982aacdce"
server_key_version: "encryptionkey@0"
I20260807 02:56:25.531035 10446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:25.561086 10446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:25.565689 10446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:25.566018 10446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:25.567004 10446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:25.568509 10446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:25.568619 10446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:25.568773 10446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:25.568868 10446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:25.605923 10446 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:45317
I20260807 02:56:25.605979 10636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:45317 every 8 connection(s)
I20260807 02:56:25.607280 10446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:56:25.611187 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 10446
I20260807 02:56:25.611459 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:56:25.612460 1509 external_mini_cluster.cc:1428] Setting key 95b6d0575b5869da31971b899fe86b2c
Aug 07 02:56:25 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071385, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:25.633957 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:25.610250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:57229 (local address 127.1.121.126:41229)
0807 02:56:25.610761 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:25.610768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:25.611854 (+ 1086us) server_negotiation.cc:407] Connection header received
0807 02:56:25.614612 (+ 2758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:25.614618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:25.614701 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:25.614814 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:25.617235 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:25.617928 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:25.620693 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:25.621190 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:25.621293 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:25.626333 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:25.626342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:25.626350 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:25.626397 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:25.630191 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:25.631004 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:25.631013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:25.631019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:25.631180 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:25.631627 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:25.631634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:25.631640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:25.632054 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:25.632253 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:25.633161 (+ 908us) server_negotiation.cc:299] Negotiation successful
0807 02:56:25.633430 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260807 02:56:25.634941 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:25.610706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.65:57229)
0807 02:56:25.611626 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:25.611712 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:25.614198 (+ 2486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:25.615221 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:25.615234 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:25.616092 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:25.617045 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:25.617078 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:25.618096 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:25.618105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:25.620510 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:25.620531 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:25.621437 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:25.621448 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:25.621823 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:25.622651 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:25.622690 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:25.626065 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:25.630355 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:25.630363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:25.630377 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:25.630860 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:25.631298 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:25.631302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:25.631304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:25.631501 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:25.632274 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:25.632281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:25.632654 (+ 373us) client_negotiation.cc:769] Sending connection context
0807 02:56:25.633006 (+ 352us) client_negotiation.cc:240] Negotiation successful
0807 02:56:25.633464 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":340,"threads_started":1}
I20260807 02:56:25.636859 10637 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41229
I20260807 02:56:25.637224 10637 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:25.638031 10637 heartbeater.cc:511] Master 127.1.121.126:41229 requested a full tablet report, sending...
I20260807 02:56:25.640563 10356 ts_manager.cc:194] Registered new tserver with Master: 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317)
WARNING: no policy specified for oryx/127.1.121.66@KRBTEST.COM; defaulting to no policy
I20260807 02:56:25.642320 10356 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:57229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab.
Entry for principal oryx/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab.
I20260807 02:56:25.680794 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41229
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab
--principal=oryx/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:26.043588 10643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:26.043975 10643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:26.044175 10643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:26.051777 10643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:26.051970 10643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:26.052165 10643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:56:26.062567 10643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab
--principal=oryx/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:26.064008 10643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:26.065703 10643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:26.080420 10650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:26.080421 10654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:26.080463 10652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:26.081593 10643 server_base.cc:1034] running on GCE node
I20260807 02:56:26.082381 10643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:26.083405 10643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:26.084697 10643 hybrid_clock.cc:648] HybridClock initialized: now 1786071386084646 us; error 66 us; skew 500 ppm
I20260807 02:56:26.085120 10643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:26 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071386, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:26.088737 10643 init.cc:377] Logged in from keytab as oryx/127.1.121.66@KRBTEST.COM (short username oryx)
I20260807 02:56:26.090327 10643 webserver.cc:460] Webserver started at http://127.1.121.66:37111/ using document root <none> and password file <none>
I20260807 02:56:26.091081 10643 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:26.091233 10643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:26.091563 10643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:26.094681 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "b64e2dc08e694040a7355cea1928fe9b"
format_stamp: "Formatted at 2026-08-07 02:56:26 on dist-test-slave-3h5h"
server_key: "0508642d7f204eb033148e877ddc1fc9"
server_key_iv: "b7ddc87b35f5f23993b2aeea255beb19"
server_key_version: "encryptionkey@0"
I20260807 02:56:26.095454 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "b64e2dc08e694040a7355cea1928fe9b"
format_stamp: "Formatted at 2026-08-07 02:56:26 on dist-test-slave-3h5h"
server_key: "0508642d7f204eb033148e877ddc1fc9"
server_key_iv: "b7ddc87b35f5f23993b2aeea255beb19"
server_key_version: "encryptionkey@0"
I20260807 02:56:26.101281 10643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 02:56:26.105615 10666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:26.107100 10643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:56:26.107293 10643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "b64e2dc08e694040a7355cea1928fe9b"
format_stamp: "Formatted at 2026-08-07 02:56:26 on dist-test-slave-3h5h"
server_key: "0508642d7f204eb033148e877ddc1fc9"
server_key_iv: "b7ddc87b35f5f23993b2aeea255beb19"
server_key_version: "encryptionkey@0"
I20260807 02:56:26.107496 10643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:26.128772 10643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:26.133158 10643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:26.133452 10643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:26.134639 10643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:26.136276 10643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:26.136358 10643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:26.136444 10643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:26.136489 10643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:26.174552 10643 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:35947
I20260807 02:56:26.174620 10822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:35947 every 8 connection(s)
I20260807 02:56:26.175904 10643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:56:26.180773 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 10643
I20260807 02:56:26.181008 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:56:26.181528 1509 external_mini_cluster.cc:1428] Setting key 2f224e07550a649a193ea4ad57f635e3
Aug 07 02:56:26 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071386, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:26.197225 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:26.178911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:50035 (local address 127.1.121.126:41229)
0807 02:56:26.179431 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:26.179439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:26.180439 (+ 1000us) server_negotiation.cc:407] Connection header received
0807 02:56:26.182563 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:26.182569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:26.182663 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:26.182797 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:26.184253 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:26.185066 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:26.186904 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:26.187336 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:26.187474 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:26.191174 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:26.191179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:26.191182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:26.191215 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:26.193994 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:26.194770 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:26.194782 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:26.194785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:26.194912 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:26.195356 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:26.195363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:26.195366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:26.195656 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:26.195824 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:26.196557 (+ 733us) server_negotiation.cc:299] Negotiation successful
0807 02:56:26.196785 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260807 02:56:26.198253 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:26.179541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.66:50035)
0807 02:56:26.180276 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:26.180321 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:26.182292 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:26.183042 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:26.183050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:26.183535 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:26.184085 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:26.184101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:26.185239 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:26.185244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:26.186736 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:26.186756 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:26.187603 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:26.187612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:26.187847 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:26.188562 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:26.188614 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:26.190924 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:26.194176 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:26.194185 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:26.194201 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:26.194635 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:26.195040 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:26.195045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:26.195049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:26.195239 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:26.195803 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:26.195811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:26.196188 (+ 377us) client_negotiation.cc:769] Sending connection context
0807 02:56:26.196467 (+ 279us) client_negotiation.cc:240] Negotiation successful
0807 02:56:26.197087 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":284,"threads_started":1}
I20260807 02:56:26.199834 10823 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41229
I20260807 02:56:26.200218 10823 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:26.201037 10823 heartbeater.cc:511] Master 127.1.121.126:41229 requested a full tablet report, sending...
I20260807 02:56:26.202661 10356 ts_manager.cc:194] Registered new tserver with Master: b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947)
I20260807 02:56:26.203999 10356 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:50035
WARNING: no policy specified for oryx/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab.
Entry for principal oryx/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab.
I20260807 02:56:26.243952 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41229
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab
--principal=oryx/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:26.585079 10830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:26.585372 10830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:26.585469 10830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:26.592818 10830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:26.592954 10830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:26.593071 10830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:56:26.603139 10830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab
--principal=oryx/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:26.604429 10830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:26.605842 10830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:26.619550 10835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:26.619647 10836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:26.619586 10838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:26.620891 10830 server_base.cc:1034] running on GCE node
I20260807 02:56:26.621609 10830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:26.622552 10830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:26.623785 10830 hybrid_clock.cc:648] HybridClock initialized: now 1786071386623738 us; error 41 us; skew 500 ppm
I20260807 02:56:26.624107 10830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:26 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071386, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:26.627247 10830 init.cc:377] Logged in from keytab as oryx/127.1.121.67@KRBTEST.COM (short username oryx)
I20260807 02:56:26.628717 10830 webserver.cc:460] Webserver started at http://127.1.121.67:38105/ using document root <none> and password file <none>
I20260807 02:56:26.629460 10830 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:26.629612 10830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:26.629927 10830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:26.632952 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "caedd08455f64c838304ade2db51fd36"
format_stamp: "Formatted at 2026-08-07 02:56:26 on dist-test-slave-3h5h"
server_key: "375ae107bbb1d0ed6f58835b69667854"
server_key_iv: "37bfcaacab25bdba6034b741ec8e40b5"
server_key_version: "encryptionkey@0"
I20260807 02:56:26.633749 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "caedd08455f64c838304ade2db51fd36"
format_stamp: "Formatted at 2026-08-07 02:56:26 on dist-test-slave-3h5h"
server_key: "375ae107bbb1d0ed6f58835b69667854"
server_key_iv: "37bfcaacab25bdba6034b741ec8e40b5"
server_key_version: "encryptionkey@0"
I20260807 02:56:26.639110 10830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:56:26.642613 10845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:26.643906 10830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:56:26.644080 10830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "caedd08455f64c838304ade2db51fd36"
format_stamp: "Formatted at 2026-08-07 02:56:26 on dist-test-slave-3h5h"
server_key: "375ae107bbb1d0ed6f58835b69667854"
server_key_iv: "37bfcaacab25bdba6034b741ec8e40b5"
server_key_version: "encryptionkey@0"
I20260807 02:56:26.644294 10830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:26.645915 10637 heartbeater.cc:503] Master 127.1.121.126:41229 was elected leader, sending a full tablet report...
I20260807 02:56:26.667966 10830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:26.672293 10830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:26.672544 10830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:26.673591 10830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:26.674973 10830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:26.675051 10830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:26.675207 10830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:26.675282 10830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:26.711474 10830 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:34967
I20260807 02:56:26.711496 11005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:34967 every 8 connection(s)
I20260807 02:56:26.712965 10830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:56:26.719518 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 10830
I20260807 02:56:26.719703 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:56:26.720187 1509 external_mini_cluster.cc:1428] Setting key 1d70cb2d919bfac74572a971434c527e
Aug 07 02:56:26 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071386, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:26.732406 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:26.715506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:56061 (local address 127.1.121.126:41229)
0807 02:56:26.716013 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:26.716020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:26.717068 (+ 1048us) server_negotiation.cc:407] Connection header received
0807 02:56:26.718877 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:26.718882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:26.718962 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:26.719061 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:26.720713 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:26.721739 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:26.723498 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:26.723896 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:26.723992 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:26.727259 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:26.727264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:26.727268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:26.727299 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:26.729817 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:26.730487 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:26.730491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:26.730493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:26.730623 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:26.730983 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:26.730987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:26.730989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:26.731235 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:26.731357 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:26.731964 (+ 607us) server_negotiation.cc:299] Negotiation successful
0807 02:56:26.732125 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":222,"threads_started":1}
I20260807 02:56:26.733439 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:26.716091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.67:56061)
0807 02:56:26.716913 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:26.716956 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:26.718588 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:26.719264 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:26.719272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:26.719900 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:26.720531 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:26.720555 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:26.721897 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:26.721902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:26.723363 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:26.723376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:26.724079 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:26.724084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:26.724276 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:26.724948 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:26.724969 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:26.727070 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:26.729944 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:26.729951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:26.729963 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:26.730386 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:26.730717 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:26.730720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:26.730723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:26.730892 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:26.731339 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:26.731345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:26.731636 (+ 291us) client_negotiation.cc:769] Sending connection context
0807 02:56:26.731877 (+ 241us) client_negotiation.cc:240] Negotiation successful
0807 02:56:26.732240 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":268,"threads_started":1}
I20260807 02:56:26.735301 11007 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41229
I20260807 02:56:26.735679 11007 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:26.736536 11007 heartbeater.cc:511] Master 127.1.121.126:41229 requested a full tablet report, sending...
I20260807 02:56:26.738098 10356 ts_manager.cc:194] Registered new tserver with Master: caedd08455f64c838304ade2db51fd36 (127.1.121.67:34967)
I20260807 02:56:26.738915 10356 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:56061
I20260807 02:56:26.748714 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.121.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.125.keytab.
Entry for principal oryx/127.1.121.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.125.keytab.
I20260807 02:56:26.813028 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.125:35089
--webserver_interface=127.1.121.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.125.keytab
--principal=oryx/127.1.121.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:41229,127.1.121.125:35089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:27.180094 11015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:27.180385 11015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:27.180594 11015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:27.189459 11015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:27.189611 11015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:27.189713 11015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:27.189827 11015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:27.189924 11015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:27.201557 11015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:41229,127.1.121.125:35089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.125.keytab
--principal=oryx/127.1.121.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.125:35089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.121.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
W20260807 02:56:27.201992 11015 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.121.126:41229,127.1.121.125:35089'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 02:56:27.203152 11015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:27.204576 11015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:56:27.206707 10823 heartbeater.cc:503] Master 127.1.121.126:41229 was elected leader, sending a full tablet report...
W20260807 02:56:27.217554 11021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:27.217684 11023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:27.217707 11025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:27.218793 11015 server_base.cc:1034] running on GCE node
I20260807 02:56:27.219596 11015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:27.221038 11015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:27.222429 11015 hybrid_clock.cc:648] HybridClock initialized: now 1786071387222349 us; error 41 us; skew 500 ppm
I20260807 02:56:27.222838 11015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:27 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071387, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:27.226263 11015 init.cc:377] Logged in from keytab as oryx/127.1.121.125@KRBTEST.COM (short username oryx)
I20260807 02:56:27.227784 11015 webserver.cc:460] Webserver started at http://127.1.121.125:40219/ using document root <none> and password file <none>
I20260807 02:56:27.228519 11015 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:27.228662 11015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:27.229060 11015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:27.232285 11015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data/instance:
uuid: "e4ad78b04e534f29b0d7fd563efb533c"
format_stamp: "Formatted at 2026-08-07 02:56:27 on dist-test-slave-3h5h"
server_key: "c993499a7902769097ca6b085b7a3cfe"
server_key_iv: "8e90b480aeccfe37f27c36bff3decdd8"
server_key_version: "encryptionkey@0"
I20260807 02:56:27.233238 11015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal/instance:
uuid: "e4ad78b04e534f29b0d7fd563efb533c"
format_stamp: "Formatted at 2026-08-07 02:56:27 on dist-test-slave-3h5h"
server_key: "c993499a7902769097ca6b085b7a3cfe"
server_key_iv: "8e90b480aeccfe37f27c36bff3decdd8"
server_key_version: "encryptionkey@0"
I20260807 02:56:27.239056 11015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:56:27.242668 11040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:27.244431 11015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:56:27.244660 11015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal
uuid: "e4ad78b04e534f29b0d7fd563efb533c"
format_stamp: "Formatted at 2026-08-07 02:56:27 on dist-test-slave-3h5h"
server_key: "c993499a7902769097ca6b085b7a3cfe"
server_key_iv: "8e90b480aeccfe37f27c36bff3decdd8"
server_key_version: "encryptionkey@0"
I20260807 02:56:27.244971 11015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:27.275076 11015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:27.280014 11015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:27.280341 11015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:27.302392 11015 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.125:35089
I20260807 02:56:27.302412 11107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.125:35089 every 8 connection(s)
I20260807 02:56:27.304234 11015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
I20260807 02:56:27.310364 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 11015
I20260807 02:56:27.310680 11109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:27.325189 11109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:56:27 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.125@KRBTEST.COM
I20260807 02:56:27.340365 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.312995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.1.121.125:35089)
0807 02:56:27.313939 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:27.313956 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:27.314013 (+ 57us) server_negotiation.cc:407] Connection header received
0807 02:56:27.316823 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:27.316890 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:27.317594 (+ 704us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:27.318163 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:27.320051 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.321812 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.323906 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.325177 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.325404 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.330645 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:27.330658 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:27.330675 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:27.330728 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:27.335118 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.335899 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.335908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.335917 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.336104 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.336530 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.336535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.336539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.337329 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:27.337656 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:27.338127 (+ 471us) server_negotiation.cc:299] Negotiation successful
0807 02:56:27.338682 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":369,"threads_started":1}
Aug 07 02:56:27 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071387, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.125@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:27.351089 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.328761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.1.121.126:41229)
0807 02:56:27.330826 (+ 2065us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:27.330843 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:27.330866 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:56:27.330933 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:27.330938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:27.331039 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:27.331151 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:27.332971 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.334191 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.336704 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.337447 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.337599 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.343196 (+ 5597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:27.343206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:27.343236 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:27.343298 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:27.346982 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.348183 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.348189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.348193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.348359 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.349001 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.349012 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.349018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.349559 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:27.349723 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:27.350457 (+ 734us) server_negotiation.cc:299] Negotiation successful
0807 02:56:27.350709 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1927,"thread_start_us":1722,"threads_started":1}
I20260807 02:56:27.351480 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.328720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41042)
0807 02:56:27.329628 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:27.329687 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:27.329871 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:27.331384 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:27.331394 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:27.332334 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:27.332747 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.332762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.334431 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.334438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.336472 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.336494 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.337870 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.337878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.338099 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.338824 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:27.338857 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:27.342948 (+ 4091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:27.347194 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.347209 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.347217 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.347911 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.348545 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.348550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.348554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.348749 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.349721 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:27.349732 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:27.350019 (+ 287us) client_negotiation.cc:769] Sending connection context
0807 02:56:27.350305 (+ 286us) client_negotiation.cc:240] Negotiation successful
0807 02:56:27.350866 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":344,"threads_started":1}
I20260807 02:56:27.353984 11109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:56:27 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071387, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.125@KRBTEST.COM for oryx/127.1.121.125@KRBTEST.COM
I20260807 02:56:27.371337 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.355054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.0.0.1:60662)
0807 02:56:27.355323 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:27.355348 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:27.355476 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:27.355897 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:27.355905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:27.356498 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:27.356728 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.356746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.357941 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.357950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.359804 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.359826 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.360577 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.360584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.360930 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.361632 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:27.361665 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:27.365127 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:27.368573 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.368578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.368583 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.369073 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.369508 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.369514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.369518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.369658 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.370237 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:27.370243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:27.370412 (+ 169us) client_negotiation.cc:769] Sending connection context
0807 02:56:27.370565 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 02:56:27.370899 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 02:56:27.371523 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.355203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.1.121.125:35089)
0807 02:56:27.355383 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:27.355390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:27.355409 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:56:27.355611 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:27.355616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:27.355714 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:27.355833 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:27.356950 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.357769 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.359977 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.360387 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.360486 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.365327 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:27.365332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:27.365336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:27.365368 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:27.368413 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.369215 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.369226 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.369228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.369378 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.369785 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.369796 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.369805 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.370107 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:27.370279 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:27.370676 (+ 397us) server_negotiation.cc:299] Negotiation successful
0807 02:56:27.371083 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":69}
I20260807 02:56:27.377570 11109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Bootstrap starting.
I20260807 02:56:27.381031 11109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:27.382186 11109 log.cc:824] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:27.385382 11109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: No bootstrap required, opened a new log
I20260807 02:56:27.393414 11109 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } }
I20260807 02:56:27.393821 11109 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:27.393931 11109 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ad78b04e534f29b0d7fd563efb533c, State: Initialized, Role: FOLLOWER
I20260807 02:56:27.394593 11109 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } }
I20260807 02:56:27.395854 11118 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } } }
I20260807 02:56:27.396306 11118 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:56:27.397199 11109 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: configured and running, proceeding with master startup.
W20260807 02:56:27.407539 11133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:56:27.407737 11133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:56:27.411890 11015 master_runner.cc:428] Detected that this master e4ad78b04e534f29b0d7fd563efb533c is joining an existing cluster
I20260807 02:56:27.412002 11015 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.121.125:35089
I20260807 02:56:27.439055 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.416188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.1.121.125:35089)
0807 02:56:27.416733 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:27.416742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:27.422583 (+ 5841us) server_negotiation.cc:407] Connection header received
0807 02:56:27.422744 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:27.422750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:27.422839 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:27.422972 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:27.424186 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.424965 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.427299 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.428202 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.428362 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.431021 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:27.431028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:27.431033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:27.431074 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:27.435453 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.436318 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.436326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.436335 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.436535 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.437055 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.437061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.437065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.437473 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:27.437662 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:27.438195 (+ 533us) server_negotiation.cc:299] Negotiation successful
0807 02:56:27.438557 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":250}
I20260807 02:56:27.439091 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.416449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.0.0.1:60678)
0807 02:56:27.422454 (+ 6005us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:27.422471 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:27.422611 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:27.423012 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:27.423016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:27.423702 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:27.424051 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.424060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.425139 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.425144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.427129 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.427146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.428441 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.428447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.428661 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.429521 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:27.429548 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:27.430806 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:27.435607 (+ 4801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.435613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.435617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.436164 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.436670 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.436678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.436682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.436897 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.437707 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:27.437710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:27.437829 (+ 119us) client_negotiation.cc:769] Sending connection context
0807 02:56:27.438045 (+ 216us) client_negotiation.cc:240] Negotiation successful
0807 02:56:27.438386 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5866,"mutex_wait_us":46,"thread_start_us":5662,"threads_started":1}
I20260807 02:56:27.439779 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.416643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.1.121.126:41229)
0807 02:56:27.416881 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:27.416890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:27.421831 (+ 4941us) server_negotiation.cc:407] Connection header received
0807 02:56:27.422057 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:27.422066 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:27.422185 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:27.422293 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:27.423870 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.425129 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.428731 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.429479 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.429619 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.432328 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:27.432336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:27.432342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:27.432393 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:27.436420 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.437266 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.437272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.437275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.437442 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.437919 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.437926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.437929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.438310 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:27.438488 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:27.439190 (+ 702us) server_negotiation.cc:299] Negotiation successful
0807 02:56:27.439407 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:56:27.440768 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.416178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41044)
0807 02:56:27.421710 (+ 5532us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:27.421728 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:27.421893 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:27.422497 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:27.422505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:27.423268 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:27.423684 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.423695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.425293 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.425299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.427129 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.427146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.429889 (+ 2743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.429896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.430146 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.430843 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:27.430866 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:27.432094 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:27.436590 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.436595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.436599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.437126 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.437600 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.437605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.437609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.437779 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.438572 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:27.438578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:27.438807 (+ 229us) client_negotiation.cc:769] Sending connection context
0807 02:56:27.439989 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0807 02:56:27.440349 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5389,"mutex_wait_us":79,"thread_start_us":5057,"threads_started":1}
Aug 07 02:56:27 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.126@KRBTEST.COM for oryx/127.1.121.125@KRBTEST.COM
I20260807 02:56:27.465857 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.447416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.1.121.125:35089)
0807 02:56:27.447628 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:27.447636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:27.448584 (+ 948us) server_negotiation.cc:407] Connection header received
0807 02:56:27.448679 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:27.448687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:27.448807 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:27.448924 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:27.450609 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.451859 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.453894 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.454719 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.454872 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.459413 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:27.459422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:27.459429 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:27.459473 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:27.462721 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.463449 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.463456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.463459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.463617 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.464042 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.464048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.464051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.464336 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:27.464510 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:27.465175 (+ 665us) server_negotiation.cc:299] Negotiation successful
0807 02:56:27.465420 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 02:56:27.465868 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.447412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.0.0.1:60680)
0807 02:56:27.448125 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:27.448170 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:27.448357 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:27.449174 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:27.449183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:27.450108 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:27.450411 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.450426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.452025 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.452032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.453700 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.453718 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.454917 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.454924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.455149 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.455932 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:27.455957 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:27.459138 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:27.462872 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.462886 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.462895 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.463320 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.463745 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.463750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.463754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.463918 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.464485 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:27.464494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:27.464751 (+ 257us) client_negotiation.cc:769] Sending connection context
0807 02:56:27.465074 (+ 323us) client_negotiation.cc:240] Negotiation successful
0807 02:56:27.465420 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":583,"threads_started":1}
I20260807 02:56:27.468065 10356 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.121.125:35089: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c"
member_type: NON_VOTER
last_known_addr {
host: "127.1.121.125"
port: 35089
}
attrs {
promote: true
}
}
dest_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9"
cas_config_opid_index: -1
I20260807 02:56:27.469432 10356 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.1.121.124@KRBTEST.COM; defaulting to no policy
I20260807 02:56:27.481186 11146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e4ad78b04e534f29b0d7fd563efb533c (127.1.121.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } } }
I20260807 02:56:27.482755 11077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.124@KRBTEST.COM" created.
I20260807 02:56:27.487083 11077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Refusing update from remote peer dfcbe99e52f743b7ab27a4fba0ccfec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 02:56:27.486675 11146 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.121.125:35089
I20260807 02:56:27.488492 11146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:27.489002 11015 master.cc:537] Master@127.1.121.125:35089 shutting down...
I20260807 02:56:27.489167 11147 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } } }
I20260807 02:56:27.493451 11147 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
W20260807 02:56:27.494551 10334 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260807 02:56:27.497450 10334 consensus_peers.cc:487] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 -> Peer e4ad78b04e534f29b0d7fd563efb533c (127.1.121.125:35089): Couldn't send request to peer e4ad78b04e534f29b0d7fd563efb533c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 02:56:27.504565 11015 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:27.505244 11015 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:27.505537 11015 tablet_replica.cc:330] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: stopping tablet replica
I20260807 02:56:27.513569 11015 master.cc:552] Master@127.1.121.125:35089 shutdown complete.
I20260807 02:56:27.513736 11015 master_runner.cc:305] Clearing existing system tablet
I20260807 02:56:27.514211 11015 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:27.514312 11015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:27.517405 11015 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 02:56:27.519934 11155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:27.520342 11015 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:27.520648 11015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:56:27.520781 11015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal
uuid: "e4ad78b04e534f29b0d7fd563efb533c"
format_stamp: "Formatted at 2026-08-07 02:56:27 on dist-test-slave-3h5h"
server_key: "c993499a7902769097ca6b085b7a3cfe"
server_key_iv: "8e90b480aeccfe37f27c36bff3decdd8"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.124.keytab.
Entry for principal oryx/127.1.121.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.124.keytab.
I20260807 02:56:27.526602 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.124:37441
--webserver_interface=127.1.121.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.124.keytab
--principal=oryx/127.1.121.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:41229,127.1.121.125:35089,127.1.121.124:37441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:56:27.526823 11015 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:27.534101 11015 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 02:56:27.534304 11015 log.cc:1196] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 02:56:27.534775 11015 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Deleting consensus metadata
I20260807 02:56:27.535159 11015 master_runner.cc:315] Copying system tablet from 127.1.121.126:41229
I20260807 02:56:27.538272 11015 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: tablet copy: Beginning tablet copy session from remote peer at address 127.1.121.126:41229
I20260807 02:56:27.557152 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.539427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.1.121.126:41229)
0807 02:56:27.539624 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:27.539631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:27.541396 (+ 1765us) server_negotiation.cc:407] Connection header received
0807 02:56:27.541716 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:27.541726 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:27.541839 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:27.541958 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:27.543001 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.544234 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.546135 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.546964 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.547111 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.549595 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:27.549605 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:27.549612 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:27.549664 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:27.553704 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.554549 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.554557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.554563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.554793 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:27.555271 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:27.555279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:27.555285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:27.555732 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:27.555916 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:27.556408 (+ 492us) server_negotiation.cc:299] Negotiation successful
0807 02:56:27.556688 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 02:56:27.557353 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:27.539331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41046)
0807 02:56:27.541275 (+ 1944us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:27.541296 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:27.541535 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:27.542014 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:27.542019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:27.542555 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:27.542824 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.542837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.544413 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.544419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.545947 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:27.545964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:27.547139 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:27.547147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:27.547331 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:27.548107 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:27.548132 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:27.549349 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:27.553862 (+ 4513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.553868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.553873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.554389 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.554927 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:27.554932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:27.554936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:27.555121 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:27.555883 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:27.555888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:27.556083 (+ 195us) client_negotiation.cc:769] Sending connection context
0807 02:56:27.556239 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 02:56:27.556586 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1796,"thread_start_us":1693,"threads_started":1}
I20260807 02:56:27.558715 10385 tablet_copy_service.cc:140] P dfcbe99e52f743b7ab27a4fba0ccfec9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e4ad78b04e534f29b0d7fd563efb533c ({username='oryx', principal='oryx/127.1.121.125@KRBTEST.COM'} at 127.0.0.1:41046)
I20260807 02:56:27.559149 10385 tablet_copy_service.cc:161] P dfcbe99e52f743b7ab27a4fba0ccfec9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e4ad78b04e534f29b0d7fd563efb533c at {username='oryx', principal='oryx/127.1.121.125@KRBTEST.COM'} at 127.0.0.1:41046: session id = e4ad78b04e534f29b0d7fd563efb533c-00000000000000000000000000000000
I20260807 02:56:27.564281 10385 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 02:56:27.567466 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 02:56:27.575014 11015 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: tablet copy: Starting download of 0 data blocks...
I20260807 02:56:27.575404 11015 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: tablet copy: Starting download of 1 WAL segments...
I20260807 02:56:27.579473 11015 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 02:56:27.584311 10385 tablet_copy_service.cc:342] P dfcbe99e52f743b7ab27a4fba0ccfec9: Request end of tablet copy session e4ad78b04e534f29b0d7fd563efb533c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.121.125@KRBTEST.COM'} at 127.0.0.1:41046
I20260807 02:56:27.584635 10385 tablet_copy_service.cc:433] P dfcbe99e52f743b7ab27a4fba0ccfec9: ending tablet copy session e4ad78b04e534f29b0d7fd563efb533c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e4ad78b04e534f29b0d7fd563efb533c
I20260807 02:56:27.599332 11015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:27.602452 11169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:27.610208 11170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:27.616982 11172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:27.617482 11015 server_base.cc:1034] running on GCE node
I20260807 02:56:27.617857 11015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:27.618551 11015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:27.619946 11015 hybrid_clock.cc:648] HybridClock initialized: now 1786071387619865 us; error 46 us; skew 500 ppm
I20260807 02:56:27.620292 11015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:27 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071387, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:27.622750 11015 init.cc:377] Logged in from keytab as oryx/127.1.121.125@KRBTEST.COM (short username oryx)
I20260807 02:56:27.626139 11015 webserver.cc:460] Webserver started at http://127.1.121.125:45563/ using document root <none> and password file <none>
I20260807 02:56:27.626565 11015 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:27.626708 11015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:27.630296 11015 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:56:27.632514 11186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:27.633006 11015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260807 02:56:27.633211 11015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal
uuid: "e4ad78b04e534f29b0d7fd563efb533c"
format_stamp: "Formatted at 2026-08-07 02:56:27 on dist-test-slave-3h5h"
server_key: "c993499a7902769097ca6b085b7a3cfe"
server_key_iv: "8e90b480aeccfe37f27c36bff3decdd8"
server_key_version: "encryptionkey@0"
I20260807 02:56:27.633541 11015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:27.646750 11015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:27.651593 11015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:27.694730 11015 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.125:35089
I20260807 02:56:27.694756 11250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.125:35089 every 8 connection(s)
I20260807 02:56:27.696405 11015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
I20260807 02:56:27.702863 11251 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:56:27.707006 11251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Bootstrap starting.
I20260807 02:56:27.730242 11251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:56:27.730860 11251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Bootstrap complete.
I20260807 02:56:27.732259 11251 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } }
I20260807 02:56:27.732606 11251 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4ad78b04e534f29b0d7fd563efb533c, State: Initialized, Role: LEARNER
I20260807 02:56:27.732960 11251 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } }
I20260807 02:56:27.733918 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } } }
I20260807 02:56:27.734367 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: This master's current role is: LEARNER
I20260807 02:56:27.734467 11251 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:27.741892 11007 heartbeater.cc:503] Master 127.1.121.126:41229 was elected leader, sending a full tablet report...
I20260807 02:56:27.743570 11273 catalog_manager.cc:1259] Loaded cluster ID: b7c877b6dc8a48c1b68079f078964abb
I20260807 02:56:27.743748 11273 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: loading cluster ID for follower catalog manager: success
I20260807 02:56:27.747479 11273 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: acquiring CA information for follower catalog manager: success
I20260807 02:56:27.750473 11273 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 02:56:27.961308 11158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:27.961748 11158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:27.961943 11158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:27.970139 11158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:27.970304 11158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:27.970425 11158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:27.970536 11158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:27.970636 11158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:27.981560 11158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:41229,127.1.121.125:35089,127.1.121.124:37441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.124.keytab
--principal=oryx/127.1.121.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.124:37441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.121.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:27.983086 11158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:27.984580 11158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:27.998456 11283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:27.998826 11281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:27.999157 11280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:28.000149 11158 server_base.cc:1034] running on GCE node
I20260807 02:56:28.001127 11158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:28.002854 11158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:28.004143 11158 hybrid_clock.cc:648] HybridClock initialized: now 1786071388004158 us; error 108 us; skew 500 ppm
I20260807 02:56:28.004602 11158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:28 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071388, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:28.008440 11158 init.cc:377] Logged in from keytab as oryx/127.1.121.124@KRBTEST.COM (short username oryx)
I20260807 02:56:28.010346 11158 webserver.cc:460] Webserver started at http://127.1.121.124:44679/ using document root <none> and password file <none>
I20260807 02:56:28.011224 11158 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:28.011379 11158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:28.011842 11158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:28.015470 11158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data/instance:
uuid: "ecd338538e82459a99489e04e9660fbf"
format_stamp: "Formatted at 2026-08-07 02:56:28 on dist-test-slave-3h5h"
server_key: "be0ccbd58715a44ad4bbc1d859fa4125"
server_key_iv: "eeb66c071d714ccad47709b00d73f4bb"
server_key_version: "encryptionkey@0"
I20260807 02:56:28.016441 11158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal/instance:
uuid: "ecd338538e82459a99489e04e9660fbf"
format_stamp: "Formatted at 2026-08-07 02:56:28 on dist-test-slave-3h5h"
server_key: "be0ccbd58715a44ad4bbc1d859fa4125"
server_key_iv: "eeb66c071d714ccad47709b00d73f4bb"
server_key_version: "encryptionkey@0"
I20260807 02:56:28.022743 11158 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 02:56:28.026945 11293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:28.028548 11158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:56:28.028761 11158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal
uuid: "ecd338538e82459a99489e04e9660fbf"
format_stamp: "Formatted at 2026-08-07 02:56:28 on dist-test-slave-3h5h"
server_key: "be0ccbd58715a44ad4bbc1d859fa4125"
server_key_iv: "eeb66c071d714ccad47709b00d73f4bb"
server_key_version: "encryptionkey@0"
I20260807 02:56:28.029048 11158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:28.062345 11158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:28.064182 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.056412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.0.0.1:60692)
0807 02:56:28.057187 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.057213 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.057366 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.057879 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.057883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.057975 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:28.058302 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.058313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.060408 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.060412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.061834 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.061848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.063565 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.063571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.063705 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.063751 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.063843 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.063944 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":295,"threads_started":1}
I20260807 02:56:28.064599 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.056766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.1.121.125:35089)
0807 02:56:28.057434 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.057440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.057463 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:56:28.057542 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.057547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.057693 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.057833 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:28.058476 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.060224 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.062034 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.063380 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.063536 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.063774 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.063892 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.064040 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":356,"threads_started":1}
I20260807 02:56:28.066107 11219 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260807 02:56:28.067067 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcbe99e52f743b7ab27a4fba0ccfec9. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } } }
I20260807 02:56:28.067371 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: This master's current role is: LEARNER
I20260807 02:56:28.068995 11158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:28.069545 11158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:28.095831 11158 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.124:37441
I20260807 02:56:28.095875 11373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.124:37441 every 8 connection(s)
I20260807 02:56:28.097489 11158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
I20260807 02:56:28.100917 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 11158
I20260807 02:56:28.102524 11374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:28.112059 11374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:56:28 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.124@KRBTEST.COM
Aug 07 02:56:28 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071388, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.124@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:28.127490 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.103541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.1.121.124:37441)
0807 02:56:28.104398 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.104410 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.104455 (+ 45us) server_negotiation.cc:407] Connection header received
0807 02:56:28.106441 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.106464 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.106865 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.107270 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.109035 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.110442 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.111923 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.112915 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.113102 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.118833 (+ 5731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.118845 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.118859 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.118903 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.122699 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.123618 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.123628 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.123635 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.123806 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.124281 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.124285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.124287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.124815 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.125157 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.125671 (+ 514us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.126107 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":385,"threads_started":1}
I20260807 02:56:28.133950 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.115388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.1.121.126:41229)
0807 02:56:28.117098 (+ 1710us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.117103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.117118 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:56:28.117168 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.117171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.117238 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.117368 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.118606 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.119884 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.121790 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.122575 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.122714 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.126715 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.126722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.126726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.126770 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.130650 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.131556 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.131561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.131563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.131714 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.132127 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.132135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.132139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.132473 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.132650 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.133281 (+ 631us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.133486 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1599,"thread_start_us":1496,"threads_started":1}
I20260807 02:56:28.134178 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.115177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41062)
0807 02:56:28.116296 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.116349 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.116562 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.117431 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.117438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.118150 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:28.118421 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.118430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.120151 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.120155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.121638 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.121652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.122807 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.122815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.123032 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.123721 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:28.123739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:28.126503 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:28.130805 (+ 4302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.130819 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.130829 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.131407 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.131820 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.131823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.131825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.131993 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.132644 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:28.132650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:28.132820 (+ 170us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.133154 (+ 334us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.133541 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":567,"threads_started":1}
I20260807 02:56:28.135898 11374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:56:28 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071388, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.124@KRBTEST.COM for oryx/127.1.121.125@KRBTEST.COM
I20260807 02:56:28.150372 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.136856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.0.0.1:60704)
0807 02:56:28.137085 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.137106 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.137275 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.137558 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.137561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.137917 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:28.138070 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.138079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.139229 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.139233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.140221 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.140234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.141089 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.141093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.141264 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.141895 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:28.141912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:28.144288 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:28.147751 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.147755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.147758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.148213 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.148714 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.148718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.148722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.149035 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.149610 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:28.149623 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:28.149754 (+ 131us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.149877 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.150139 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 02:56:28.150797 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.137064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60704 (local address 127.1.121.125:35089)
0807 02:56:28.137274 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.137279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.137298 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:56:28.137371 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.137375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.137438 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.137528 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.138205 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.139066 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.140353 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.140892 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.141022 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.144492 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.144498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.144501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.144535 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.147605 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.148396 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.148401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.148403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.148565 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.149153 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.149157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.149159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.149448 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.149571 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.150128 (+ 557us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.150334 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 02:56:28.152096 11374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:56:28 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071388, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.124@KRBTEST.COM for oryx/127.1.121.124@KRBTEST.COM
I20260807 02:56:28.166713 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.152945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.1.121.124:37441)
0807 02:56:28.153143 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.153150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.153357 (+ 207us) server_negotiation.cc:407] Connection header received
0807 02:56:28.153554 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.153559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.153624 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.153731 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.154477 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.155469 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.156655 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.157364 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.157476 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.161176 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.161181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.161184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.161215 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.163814 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.164495 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.164499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.164502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.164626 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.165317 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.165321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.165323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.165682 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.165794 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.166212 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.166404 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:56:28.166720 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.153076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:37441 (local address 127.0.0.1:60598)
0807 02:56:28.153267 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.153293 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.153388 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.153789 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.153793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.154184 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:28.154331 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.154339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.155596 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.155600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.156500 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.156512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.157560 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.157565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.157727 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.158430 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:28.158449 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:28.160920 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:28.163942 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.163946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.163948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.164383 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.164767 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.164771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.164774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.165195 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.165820 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:28.165824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:28.165966 (+ 142us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.166087 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.166277 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 02:56:28.172509 11374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Bootstrap starting.
I20260807 02:56:28.177317 11374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:28.178747 11374 log.cc:824] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:28.182062 11374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: No bootstrap required, opened a new log
I20260807 02:56:28.190048 11374 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } }
I20260807 02:56:28.190385 11374 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:28.190567 11374 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd338538e82459a99489e04e9660fbf, State: Initialized, Role: FOLLOWER
I20260807 02:56:28.191242 11374 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } }
I20260807 02:56:28.192281 11382 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } } }
I20260807 02:56:28.192662 11382 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:56:28.193598 11374 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:28.197693 11382 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:28.198208 11382 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } }
I20260807 02:56:28.201570 11219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecd338538e82459a99489e04e9660fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ad78b04e534f29b0d7fd563efb533c" is_pre_election: true
I20260807 02:56:28.201576 10369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecd338538e82459a99489e04e9660fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" is_pre_election: true
I20260807 02:56:28.203174 11382 leader_election.cc:288] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcbe99e52f743b7ab27a4fba0ccfec9 (127.1.121.126:41229), e4ad78b04e534f29b0d7fd563efb533c (127.1.121.125:35089)
I20260807 02:56:28.203197 11294 leader_election.cc:302] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecd338538e82459a99489e04e9660fbf; no voters: dfcbe99e52f743b7ab27a4fba0ccfec9, e4ad78b04e534f29b0d7fd563efb533c
I20260807 02:56:28.203796 11382 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 02:56:28.208031 11397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:56:28.208204 11397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:56:28.212433 11158 master_runner.cc:428] Detected that this master ecd338538e82459a99489e04e9660fbf is joining an existing cluster
I20260807 02:56:28.212659 11158 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.121.124:37441
I20260807 02:56:28.235008 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 10446
I20260807 02:56:28.248423 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.218416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.1.121.126:41229)
0807 02:56:28.218766 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.218775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.226875 (+ 8100us) server_negotiation.cc:407] Connection header received
0807 02:56:28.227800 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.227811 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.227990 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.228135 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.230472 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.232263 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.236099 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.236961 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.237098 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.240249 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.240258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.240262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.240311 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.244416 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.245422 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.245430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.245435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.245633 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.246238 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.246247 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.246251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.246656 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.246842 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.247700 (+ 858us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.247986 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260807 02:56:28.249866 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.218862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41070)
0807 02:56:28.226707 (+ 7845us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.226732 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.227619 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.228408 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.228416 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.229600 (+ 1184us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:28.230210 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.230234 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.233060 (+ 2826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.233070 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.235658 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.235675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.237434 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.237442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.237628 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.238539 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:28.238572 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:28.240013 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:28.244608 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.244613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.244617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.245261 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.245827 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.245835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.245841 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.246070 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.246959 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:28.246968 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:28.247314 (+ 346us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.248785 (+ 1471us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.249230 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7666,"thread_start_us":7476,"threads_started":1}
I20260807 02:56:28.254552 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.219083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.1.121.125:35089)
0807 02:56:28.219439 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.219445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.234374 (+ 14929us) server_negotiation.cc:407] Connection header received
0807 02:56:28.234693 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.234700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.234790 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.234867 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.236755 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.238079 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.240965 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.241751 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.241864 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.244878 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.244888 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.244893 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.244937 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.249492 (+ 4555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.251199 (+ 1707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.251205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.251208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.251378 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.252299 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.252306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.252311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.252760 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.252938 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.253725 (+ 787us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.254032 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260807 02:56:28.256032 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.219704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.0.0.1:60720)
0807 02:56:28.234256 (+ 14552us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.234280 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.234509 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.235290 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.235304 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.236244 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:28.236561 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.236576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.238343 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.238350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.240740 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.240760 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.242207 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.242218 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.242463 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.243250 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:28.243280 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:28.244610 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:28.249683 (+ 5073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.249688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.249692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.251037 (+ 1345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.251566 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.251571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.251575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.252127 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.253068 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:28.253075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:28.253323 (+ 248us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.255083 (+ 1760us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.255492 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14022,"thread_start_us":14203,"threads_started":1}
I20260807 02:56:28.263958 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.218870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.1.121.124:37441)
0807 02:56:28.219172 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.219183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.243186 (+ 24003us) server_negotiation.cc:407] Connection header received
0807 02:56:28.243379 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.243387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.243480 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.243586 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.245724 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.247091 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.249681 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.250388 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.250489 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.254611 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.254625 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.254631 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.254681 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.259991 (+ 5310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.261657 (+ 1666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.261662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.261665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.261824 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.262246 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.262253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.262256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.262607 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.262757 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.263357 (+ 600us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.263579 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":742656}
I20260807 02:56:28.265555 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.219961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:37441 (local address 127.0.0.1:60600)
0807 02:56:28.243063 (+ 23102us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.243086 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.243261 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.243855 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.243862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.244962 (+ 1100us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:28.245527 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.245544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.247313 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.247318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.249512 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.249531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.251047 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.251054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.251232 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.252445 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:28.252469 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:28.254207 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:28.260984 (+ 6777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.260989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.260993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.261516 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.261963 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.261972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.261975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.262134 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.262824 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:28.262828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:28.263009 (+ 181us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.264750 (+ 1741us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.265059 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22959,"thread_start_us":22802,"threads_started":1}
Aug 07 02:56:28 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.65@KRBTEST.COM
Aug 07 02:56:28 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.126@KRBTEST.COM for oryx/127.1.121.124@KRBTEST.COM
I20260807 02:56:28.278779 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.236732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.1.121.65:45317)
0807 02:56:28.249907 (+ 13175us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.249918 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.249951 (+ 33us) server_negotiation.cc:407] Connection header received
0807 02:56:28.250039 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.250049 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.250375 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.251583 (+ 1208us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.253644 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.255376 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.258085 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.259121 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.259855 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.270701 (+ 10846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.270724 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.270745 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.270801 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.275219 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.276014 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.276023 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.276027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.276201 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.276725 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.276730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.276734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.277230 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.277551 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.278018 (+ 467us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.278325 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12965,"thread_start_us":13767,"threads_started":1}
I20260807 02:56:28.280702 10631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:28.283154 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.256580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:37441 (local address 127.0.0.1:60608)
0807 02:56:28.256934 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.256955 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.257063 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.264659 (+ 7596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.264665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.265362 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:28.265673 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.265693 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.268111 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.268119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.269585 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.269602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.270883 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.270890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.271092 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.271797 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:28.271824 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:28.274243 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:28.279244 (+ 5001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.279250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.279254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.279749 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.280166 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.280171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.280175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.280331 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.281148 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:28.281157 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:28.281458 (+ 301us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.281662 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.282124 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260807 02:56:28.285095 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.256990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.1.121.124:37441)
0807 02:56:28.264262 (+ 7272us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.264268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.264284 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:56:28.264380 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.264385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.264469 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.264576 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.266226 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.267940 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.269773 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.270704 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.270860 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.274460 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.274465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.274469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.274503 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.279045 (+ 4542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.279893 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.279901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.279910 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.280040 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.280473 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.280479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.280481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.280791 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.281083 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.283784 (+ 2701us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.284412 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7166,"thread_start_us":10716,"threads_started":1}
I20260807 02:56:28.286742 10356 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.121.124:37441: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ecd338538e82459a99489e04e9660fbf"
member_type: NON_VOTER
last_known_addr {
host: "127.1.121.124"
port: 37441
}
attrs {
promote: true
}
}
dest_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9"
cas_config_opid_index: 5
I20260807 02:56:28.287942 10356 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } }
I20260807 02:56:28.290169 11300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ecd338538e82459a99489e04e9660fbf (127.1.121.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.290935 11219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 LEARNER]: Refusing update from remote peer dfcbe99e52f743b7ab27a4fba0ccfec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 02:56:28.295609 11339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:28.300491 11339 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 FOLLOWER]: Refusing update from remote peer dfcbe99e52f743b7ab27a4fba0ccfec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 02:56:28.301803 11300 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.121.124:37441
I20260807 02:56:28.302508 11158 master.cc:537] Master@127.1.121.124:37441 shutting down...
I20260807 02:56:28.313606 11413 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260807 02:56:28.316507 11412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:28.325363 11219 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ecd338538e82459a99489e04e9660fbf (127.1.121.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.328116 11414 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.328483 11414 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:28.330901 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: NON_VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: true } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.331203 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: This master's current role is: LEARNER
I20260807 02:56:28.331799 11412 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: attempting to promote NON_VOTER e4ad78b04e534f29b0d7fd563efb533c to VOTER
I20260807 02:56:28.332712 11412 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } }
I20260807 02:56:28.335494 11219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 LEARNER]: Refusing update from remote peer dfcbe99e52f743b7ab27a4fba0ccfec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 02:56:28.336165 11412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 02:56:28.338822 11414 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4ad78b04e534f29b0d7fd563efb533c (127.1.121.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
W20260807 02:56:28.341882 11306 tablet.cc:2369] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:56:28.343472 11414 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.344177 11418 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.345425 11418 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:28.345513 11158 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:28.345693 11414 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:28.345932 11158 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:28.346168 11158 tablet_replica.cc:330] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: stopping tablet replica
I20260807 02:56:28.346359 11219 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4ad78b04e534f29b0d7fd563efb533c (127.1.121.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.351504 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.351855 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:56:28.355348 11158 master.cc:552] Master@127.1.121.124:37441 shutdown complete.
I20260807 02:56:28.355490 11158 master_runner.cc:305] Clearing existing system tablet
I20260807 02:56:28.355963 11158 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:28.356146 11158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:28.366406 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.365875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:37441 (local address 127.0.0.1:60612)
0807 02:56:28.366051 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.366198 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 02:56:28.366750 10335 consensus_peers.cc:487] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 -> Peer ecd338538e82459a99489e04e9660fbf (127.1.121.124:37441): Couldn't send request to peer ecd338538e82459a99489e04e9660fbf. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.124:37441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 02:56:28.367654 11158 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.002s sys 0.008s
I20260807 02:56:28.369807 11423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:28.370214 11158 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:28.370512 11158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:56:28.370627 11158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal
uuid: "ecd338538e82459a99489e04e9660fbf"
format_stamp: "Formatted at 2026-08-07 02:56:28 on dist-test-slave-3h5h"
server_key: "be0ccbd58715a44ad4bbc1d859fa4125"
server_key_iv: "eeb66c071d714ccad47709b00d73f4bb"
server_key_version: "encryptionkey@0"
I20260807 02:56:28.376286 11158 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:28.384120 11158 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 02:56:28.384298 11158 log.cc:1196] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 02:56:28.384922 11158 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Deleting consensus metadata
I20260807 02:56:28.385280 11158 master_runner.cc:315] Copying system tablet from 127.1.121.126:41229
I20260807 02:56:28.390901 11158 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: tablet copy: Beginning tablet copy session from remote peer at address 127.1.121.126:41229
I20260807 02:56:28.410552 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.391779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41080)
0807 02:56:28.393420 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.393443 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.393627 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.394112 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.394119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.394850 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:28.395218 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.395234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.396875 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.396882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.398894 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.398911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.400114 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.400121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.400362 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.401225 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:28.401252 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:28.402533 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:28.407067 (+ 4534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.407074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.407079 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.407818 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.408305 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:28.408311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:28.408315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:28.408527 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:28.409320 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:28.409325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:28.409518 (+ 193us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.409675 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.409998 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"thread_start_us":1270,"threads_started":1}
I20260807 02:56:28.410699 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.392275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.1.121.126:41229)
0807 02:56:28.392534 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.392541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.393569 (+ 1028us) server_negotiation.cc:407] Connection header received
0807 02:56:28.393839 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.393845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.393931 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.394049 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:28.395402 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.396651 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.399070 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.399911 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.400074 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.402805 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:28.402812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:28.402817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:28.402860 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:28.406897 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.407993 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.407999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.408003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.408172 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:28.408676 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:28.408681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:28.408685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:28.409102 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:28.409251 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.409890 (+ 639us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.410171 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260807 02:56:28.411875 10385 tablet_copy_service.cc:140] P dfcbe99e52f743b7ab27a4fba0ccfec9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ecd338538e82459a99489e04e9660fbf ({username='oryx', principal='oryx/127.1.121.124@KRBTEST.COM'} at 127.0.0.1:41080)
I20260807 02:56:28.412075 10385 tablet_copy_service.cc:161] P dfcbe99e52f743b7ab27a4fba0ccfec9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ecd338538e82459a99489e04e9660fbf at {username='oryx', principal='oryx/127.1.121.124@KRBTEST.COM'} at 127.0.0.1:41080: session id = ecd338538e82459a99489e04e9660fbf-00000000000000000000000000000000
I20260807 02:56:28.416401 10385 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 02:56:28.418666 11158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 02:56:28.426581 11158 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: tablet copy: Starting download of 0 data blocks...
I20260807 02:56:28.426961 11158 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: tablet copy: Starting download of 1 WAL segments...
I20260807 02:56:28.427183 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 10446
I20260807 02:56:28.434463 11158 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 02:56:28.439867 10385 tablet_copy_service.cc:342] P dfcbe99e52f743b7ab27a4fba0ccfec9: Request end of tablet copy session ecd338538e82459a99489e04e9660fbf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.121.124@KRBTEST.COM'} at 127.0.0.1:41080
I20260807 02:56:28.440058 10385 tablet_copy_service.cc:433] P dfcbe99e52f743b7ab27a4fba0ccfec9: ending tablet copy session ecd338538e82459a99489e04e9660fbf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ecd338538e82459a99489e04e9660fbf
I20260807 02:56:28.454190 11158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:28.457985 11432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:28.458034 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:45317
--local_ip_for_outbound_sockets=127.1.121.65
--tserver_master_addrs=127.1.121.126:41229
--webserver_port=46675
--webserver_interface=127.1.121.65
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab
--principal=oryx/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.121.126:41229,127.1.121.125:35089
--tserver_master_addrs=127.1.121.126:41229,127.1.121.125:35089,127.1.121.124:37441 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:28.461663 11430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:28.472147 11439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:28.472186 11158 server_base.cc:1034] running on GCE node
I20260807 02:56:28.472642 11158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:28.473443 11158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:28.474655 11158 hybrid_clock.cc:648] HybridClock initialized: now 1786071388474612 us; error 33 us; skew 500 ppm
I20260807 02:56:28.474902 11158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:28 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071388, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:28.476941 11158 init.cc:377] Logged in from keytab as oryx/127.1.121.124@KRBTEST.COM (short username oryx)
I20260807 02:56:28.483453 11158 webserver.cc:460] Webserver started at http://127.1.121.124:38277/ using document root <none> and password file <none>
I20260807 02:56:28.483798 11158 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:28.483899 11158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:28.487288 11158 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260807 02:56:28.489466 11449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:28.490008 11158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:56:28.490186 11158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal
uuid: "ecd338538e82459a99489e04e9660fbf"
format_stamp: "Formatted at 2026-08-07 02:56:28 on dist-test-slave-3h5h"
server_key: "be0ccbd58715a44ad4bbc1d859fa4125"
server_key_iv: "eeb66c071d714ccad47709b00d73f4bb"
server_key_version: "encryptionkey@0"
I20260807 02:56:28.490439 11158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:28.510748 11158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:28.515038 11158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:28.577643 11158 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.124:37441
I20260807 02:56:28.577733 11517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.124:37441 every 8 connection(s)
I20260807 02:56:28.578780 11158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
I20260807 02:56:28.584942 11518 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:56:28.589946 11518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Bootstrap starting.
I20260807 02:56:28.612700 11518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:56:28.613464 11518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: Bootstrap complete.
I20260807 02:56:28.614887 11518 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } }
I20260807 02:56:28.615247 11518 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ecd338538e82459a99489e04e9660fbf, State: Initialized, Role: LEARNER
I20260807 02:56:28.615571 11518 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } }
I20260807 02:56:28.616398 11522 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.616731 11518 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:28.616895 11522 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: This master's current role is: LEARNER
I20260807 02:56:28.629945 11537 catalog_manager.cc:1259] Loaded cluster ID: b7c877b6dc8a48c1b68079f078964abb
I20260807 02:56:28.630056 11537 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: loading cluster ID for follower catalog manager: success
I20260807 02:56:28.633548 11537 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: acquiring CA information for follower catalog manager: success
I20260807 02:56:28.636147 11537 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 02:56:28.891413 11440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:28.891817 11440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:28.891999 11440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:28.901108 11440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:28.901289 11440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:28.901487 11440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:56:28.913836 11440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab
--principal=oryx/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:45317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=46675
--tserver_master_addrs=127.1.121.126:41229,127.1.121.125:35089,127.1.121.124:37441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:28.915364 11440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:28.916929 11440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:28.932652 11545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:28.932699 11548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:28.937070 11543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:28.940127 11440 server_base.cc:1034] running on GCE node
I20260807 02:56:28.940981 11440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:28.942222 11440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:28.943476 11440 hybrid_clock.cc:648] HybridClock initialized: now 1786071388943399 us; error 67 us; skew 500 ppm
I20260807 02:56:28.943898 11440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:28 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071388, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:28.948524 11440 init.cc:377] Logged in from keytab as oryx/127.1.121.65@KRBTEST.COM (short username oryx)
I20260807 02:56:28.950549 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.939246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:37441 (local address 127.0.0.1:45894)
0807 02:56:28.942754 (+ 3508us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:28.942770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:28.942888 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:28.943315 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:28.943318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:28.943340 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:28.943692 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.943711 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.945699 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.945705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.947658 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:28.947671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.949560 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.949572 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:28.949888 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.949924 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 02:56:28.950050 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 02:56:28.950190 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3389,"thread_start_us":3276,"threads_started":1}
I20260807 02:56:28.950595 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:28.939486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.1.121.124:37441)
0807 02:56:28.940226 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:28.940236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:28.942868 (+ 2632us) server_negotiation.cc:407] Connection header received
0807 02:56:28.943048 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:28.943054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:28.943171 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:28.943296 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:28.944029 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.945545 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.947838 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:28.949252 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:28.949380 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:28.949623 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:28.950099 (+ 476us) server_negotiation.cc:299] Negotiation successful
0807 02:56:28.950255 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":383,"threads_started":1}
I20260807 02:56:28.951303 11440 webserver.cc:460] Webserver started at http://127.1.121.65:46675/ using document root <none> and password file <none>
I20260807 02:56:28.951792 11485 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260807 02:56:28.952167 11440 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:28.952346 11440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:28.952450 11522 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcbe99e52f743b7ab27a4fba0ccfec9. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:28.952806 11522 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: This master's current role is: LEARNER
I20260807 02:56:28.958613 11440 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260807 02:56:28.962042 11561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:28.963392 11440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:56:28.963562 11440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "313d60cfddb645daaf6e649f261168c5"
format_stamp: "Formatted at 2026-08-07 02:56:25 on dist-test-slave-3h5h"
server_key: "bf9cfa7d717243f01bbd31a3b5c24106"
server_key_iv: "6ad14d38a2b17b28a151351982aacdce"
server_key_version: "encryptionkey@0"
I20260807 02:56:28.964731 11440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:28.988976 11440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:28.994222 11440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:28.994557 11440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:28.995779 11440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:28.997509 11440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:28.997601 11440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:28.997749 11440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:28.997843 11440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:29.045151 11440 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:45317
I20260807 02:56:29.045241 11723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:45317 every 8 connection(s)
I20260807 02:56:29.046700 11440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:56:29.056536 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 11440
I20260807 02:56:29.056993 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 10643
Aug 07 02:56:29 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071388, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.124@KRBTEST.COM
Aug 07 02:56:29 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071388, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.125@KRBTEST.COM
Aug 07 02:56:29 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071388, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:29.079694 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.050224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:51049 (local address 127.1.121.125:35089)
0807 02:56:29.053730 (+ 3506us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:29.053737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:29.053770 (+ 33us) server_negotiation.cc:407] Connection header received
0807 02:56:29.055655 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:29.055661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:29.055770 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:29.055878 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:29.060373 (+ 4495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.061656 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.063932 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.064682 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.064824 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.071280 (+ 6456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:29.071289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:29.071295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:29.071348 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:29.076156 (+ 4808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.077239 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.077248 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.077251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.077399 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.077886 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.077892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.077897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.078321 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:29.078490 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:29.078951 (+ 461us) server_negotiation.cc:299] Negotiation successful
0807 02:56:29.079194 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3302,"thread_start_us":3182,"threads_started":1}
Aug 07 02:56:29 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.66@KRBTEST.COM
I20260807 02:56:29.080524 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.050908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:37441 (local address 127.1.121.65:33097)
0807 02:56:29.052611 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:29.052676 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:29.055223 (+ 2547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:29.056208 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:29.056222 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:29.057365 (+ 1143us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:29.058462 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.058499 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.059833 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.059842 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.062915 (+ 3073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.062933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.064200 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.064208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.064595 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.065578 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:29.065614 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:29.069847 (+ 4233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:29.074999 (+ 5152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.075008 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.075028 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.075752 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.076285 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.076290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.076294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.076591 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.077418 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:29.077427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:29.077926 (+ 499us) client_negotiation.cc:769] Sending connection context
0807 02:56:29.078275 (+ 349us) client_negotiation.cc:240] Negotiation successful
0807 02:56:29.078877 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"mutex_wait_us":155,"thread_start_us":2729,"threads_started":1}
I20260807 02:56:29.081377 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.050908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.65:33961)
0807 02:56:29.054095 (+ 3187us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:29.054110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:29.055233 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:29.056746 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:29.056752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:29.057696 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:29.058901 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.058914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.063162 (+ 4248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.063168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.065889 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.065907 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.067019 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.067023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.067160 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.068975 (+ 1815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:29.068999 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:29.071749 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:29.077239 (+ 5490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.077246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.077253 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.077811 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.078241 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.078248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.078251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.078427 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.080201 (+ 1774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:29.080206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:29.080381 (+ 175us) client_negotiation.cc:769] Sending connection context
0807 02:56:29.080548 (+ 167us) client_negotiation.cc:240] Negotiation successful
0807 02:56:29.080890 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3029,"mutex_wait_us":1328,"thread_start_us":501,"threads_started":1}
I20260807 02:56:29.081710 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.051049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:33097 (local address 127.1.121.124:37441)
0807 02:56:29.051339 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:29.051350 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:29.052811 (+ 1461us) server_negotiation.cc:407] Connection header received
0807 02:56:29.055593 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:29.055602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:29.055702 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:29.055786 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:29.058697 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.059654 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.063124 (+ 3470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.063923 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.064101 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.070193 (+ 6092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:29.070204 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:29.070212 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:29.070264 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:29.074787 (+ 4523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.075935 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.075943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.075949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.076132 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.076746 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.076753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.076759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.077194 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:29.077344 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:29.081073 (+ 3729us) server_negotiation.cc:299] Negotiation successful
0807 02:56:29.081346 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 02:56:29.082543 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.050855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:33961 (local address 127.1.121.126:41229)
0807 02:56:29.051878 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:29.051887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:29.054215 (+ 2328us) server_negotiation.cc:407] Connection header received
0807 02:56:29.055536 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:29.055541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:29.055644 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:29.055768 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:29.059549 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.061132 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.066088 (+ 4956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.066698 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.066779 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.072342 (+ 5563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:29.072358 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:29.072366 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:29.072420 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:29.077115 (+ 4695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.077941 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.077948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.077954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.078123 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.079328 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.079334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.079338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.079768 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:29.080256 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:29.081419 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0807 02:56:29.082004 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":747,"threads_started":1}
I20260807 02:56:29.082867 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.051290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.1.121.65:51049)
0807 02:56:29.053098 (+ 1808us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:29.053117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:29.055223 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:29.056208 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:29.056222 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:29.057612 (+ 1390us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:29.060119 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.060133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.061871 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.061877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.063718 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.063738 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.065130 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.065140 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.065358 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.066043 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:29.066077 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:29.071030 (+ 4953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:29.076368 (+ 5338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.076375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.076379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.077054 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.077558 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.077563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.077567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.077746 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.078463 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:29.078468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:29.078638 (+ 170us) client_negotiation.cc:769] Sending connection context
0807 02:56:29.078810 (+ 172us) client_negotiation.cc:240] Negotiation successful
0807 02:56:29.079096 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1663,"mutex_wait_us":1766,"spinlock_wait_cycles":37632,"thread_start_us":1827,"threads_started":1}
I20260807 02:56:29.083142 11724 heartbeater.cc:347] Connected to a master server at 127.1.121.124:37441
I20260807 02:56:29.083504 11724 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:29.083730 11725 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41229
I20260807 02:56:29.083894 11725 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:29.083864 11726 heartbeater.cc:347] Connected to a master server at 127.1.121.125:35089
I20260807 02:56:29.084064 11726 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:29.085163 11725 heartbeater.cc:511] Master 127.1.121.126:41229 requested a full tablet report, sending...
I20260807 02:56:29.085309 11724 heartbeater.cc:511] Master 127.1.121.124:37441 requested a full tablet report, sending...
I20260807 02:56:29.085412 11726 heartbeater.cc:511] Master 127.1.121.125:35089 requested a full tablet report, sending...
I20260807 02:56:29.087484 10356 ts_manager.cc:194] Re-registered known tserver with Master: 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317)
I20260807 02:56:29.088020 11472 ts_manager.cc:194] Registered new tserver with Master: 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317)
I20260807 02:56:29.088186 11207 ts_manager.cc:194] Registered new tserver with Master: 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317)
I20260807 02:56:29.088796 10356 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:33961
I20260807 02:56:29.090500 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.058621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.1.121.66:35947)
0807 02:56:29.060445 (+ 1824us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:29.060458 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:29.064169 (+ 3711us) server_negotiation.cc:407] Connection header received
0807 02:56:29.064517 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:29.064528 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:29.064867 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:29.065051 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:29.067889 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.069547 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.072400 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.073423 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.073539 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.081164 (+ 7625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:29.081181 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:29.081200 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:29.081251 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:29.086546 (+ 5295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.087544 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.087553 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.087556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.087744 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.088231 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.088236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.088239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.088715 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:29.089030 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:29.089730 (+ 700us) server_negotiation.cc:299] Negotiation successful
0807 02:56:29.090005 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":2628,"threads_started":1}
I20260807 02:56:29.091826 10818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:29.177801 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 10643
I20260807 02:56:29.201787 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:35947
--local_ip_for_outbound_sockets=127.1.121.66
--tserver_master_addrs=127.1.121.126:41229
--webserver_port=37111
--webserver_interface=127.1.121.66
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab
--principal=oryx/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.121.126:41229,127.1.121.125:35089
--tserver_master_addrs=127.1.121.126:41229,127.1.121.125:35089,127.1.121.124:37441 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:56:29.512253 11551 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: NON_VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: true } } }
I20260807 02:56:29.512787 11551 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:29.514279 11551 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: attempting to promote NON_VOTER ecd338538e82459a99489e04e9660fbf to VOTER
I20260807 02:56:29.514950 11551 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } }
I20260807 02:56:29.517508 11485 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 LEARNER]: Refusing update from remote peer dfcbe99e52f743b7ab27a4fba0ccfec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 02:56:29.518010 11219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Refusing update from remote peer dfcbe99e52f743b7ab27a4fba0ccfec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 02:56:29.518225 11551 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 02:56:29.518821 11743 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 02:56:29.523022 11551 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ecd338538e82459a99489e04e9660fbf (127.1.121.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:29.524072 11219 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ecd338538e82459a99489e04e9660fbf (127.1.121.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:29.527050 11551 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:29.527321 11551 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:29.527602 11743 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:29.527873 11743 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:29.531235 11743 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:29.531648 11743 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:29.533344 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:29.533746 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:56:29.533123 11485 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ecd338538e82459a99489e04e9660fbf (127.1.121.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:29.537930 11752 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:29.538388 11752 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: This master's current role is: FOLLOWER
W20260807 02:56:29.569609 11740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:29.569943 11740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:29.570124 11740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:29.577786 11740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:29.577946 11740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:29.578121 11740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:56:29.589279 11740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab
--principal=oryx/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:35947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=37111
--tserver_master_addrs=127.1.121.126:41229,127.1.121.125:35089,127.1.121.124:37441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:29.590834 11740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:29.592338 11740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:29.606829 11763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:29.606860 11764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:29.607139 11766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:29.607822 11740 server_base.cc:1034] running on GCE node
I20260807 02:56:29.608528 11740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:29.609648 11740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:29.610867 11740 hybrid_clock.cc:648] HybridClock initialized: now 1786071389610800 us; error 58 us; skew 500 ppm
I20260807 02:56:29.611254 11740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:29 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071389, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:29.614701 11740 init.cc:377] Logged in from keytab as oryx/127.1.121.66@KRBTEST.COM (short username oryx)
I20260807 02:56:29.616156 11740 webserver.cc:460] Webserver started at http://127.1.121.66:37111/ using document root <none> and password file <none>
I20260807 02:56:29.617148 11740 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:29.617296 11740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:29.622990 11740 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260807 02:56:29.625718 11775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:29.627177 11740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:29.627364 11740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "b64e2dc08e694040a7355cea1928fe9b"
format_stamp: "Formatted at 2026-08-07 02:56:26 on dist-test-slave-3h5h"
server_key: "0508642d7f204eb033148e877ddc1fc9"
server_key_iv: "b7ddc87b35f5f23993b2aeea255beb19"
server_key_version: "encryptionkey@0"
I20260807 02:56:29.628378 11740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:29.648512 11740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:29.653144 11740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:29.653446 11740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:29.654527 11740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:29.655988 11740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:29.656126 11740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:29.656257 11740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:29.656353 11740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:29.690325 11740 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:35947
I20260807 02:56:29.690379 11937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:35947 every 8 connection(s)
I20260807 02:56:29.692091 11740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:56:29.701581 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 11740
I20260807 02:56:29.701869 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 10830
Aug 07 02:56:29 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071389, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
Aug 07 02:56:29 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071389, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.125@KRBTEST.COM
Aug 07 02:56:29 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071389, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.124@KRBTEST.COM
Aug 07 02:56:29 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.67@KRBTEST.COM
I20260807 02:56:29.745434 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.695071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:56145 (local address 127.1.121.126:41229)
0807 02:56:29.695893 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:29.695900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:29.720868 (+ 24968us) server_negotiation.cc:407] Connection header received
0807 02:56:29.722941 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:29.722949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:29.723062 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:29.723196 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:29.726053 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.727293 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.729721 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.730383 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.730508 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.736762 (+ 6254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:29.736771 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:29.736777 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:29.736860 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:29.741129 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.742319 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.742324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.742328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.742503 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.743145 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.743149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.743152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.743503 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:29.743656 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:29.744740 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0807 02:56:29.745014 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":764,"threads_started":1}
I20260807 02:56:29.745434 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.695631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:45197 (local address 127.1.121.124:37441)
0807 02:56:29.696074 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:29.696080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:29.719207 (+ 23127us) server_negotiation.cc:407] Connection header received
0807 02:56:29.723489 (+ 4282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:29.723496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:29.723590 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:29.723679 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:29.725935 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.727296 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.730738 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.731417 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.731559 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.737331 (+ 5772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:29.737339 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:29.737343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:29.737387 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:29.741351 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.742319 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.742324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.742328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.742503 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.743162 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.743167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.743170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.743501 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:29.743648 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:29.744757 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0807 02:56:29.745014 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":298,"threads_started":1}
I20260807 02:56:29.745452 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.695059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:47817 (local address 127.1.121.125:35089)
0807 02:56:29.695545 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:29.695552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:29.721420 (+ 25868us) server_negotiation.cc:407] Connection header received
0807 02:56:29.722953 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:29.722960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:29.723057 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:29.723132 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:29.726367 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.727485 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.730883 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.731575 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.731714 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.736956 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:29.736966 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:29.736973 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:29.737029 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:29.741408 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.742523 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.742531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.742538 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.742719 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.743265 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.743273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.743278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.743694 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:29.743843 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:29.744767 (+ 924us) server_negotiation.cc:299] Negotiation successful
0807 02:56:29.745065 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":399,"threads_started":1}
I20260807 02:56:29.746901 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.703304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.1.121.67:34967)
0807 02:56:29.721708 (+ 18404us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:29.721716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:29.721739 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:56:29.721797 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:29.721804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:29.722013 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:29.722147 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:29.724670 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.727208 (+ 2538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.729316 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.730105 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.730259 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.738214 (+ 7955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:29.738232 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:29.738251 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:29.738305 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:29.742480 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.743393 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.743403 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.743406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.743591 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:29.744131 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:29.744138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:29.744142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:29.744588 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:29.744908 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:29.746155 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0807 02:56:29.746437 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18199,"thread_start_us":18050,"threads_started":1}
I20260807 02:56:29.747861 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.696578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:37441 (local address 127.1.121.66:45197)
0807 02:56:29.718940 (+ 22362us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:29.719022 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:29.722604 (+ 3582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:29.723944 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:29.723949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:29.724898 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:29.725749 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.725781 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.727507 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.727515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.729556 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.729571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.731738 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.731745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.731976 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.734531 (+ 2555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:29.734558 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:29.737083 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:29.741505 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.741511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.741515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.742172 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.742644 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.742649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.742652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.743005 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.743728 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:29.743737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:29.744269 (+ 532us) client_negotiation.cc:769] Sending connection context
0807 02:56:29.745675 (+ 1406us) client_negotiation.cc:240] Negotiation successful
0807 02:56:29.746419 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5552,"mutex_wait_us":332,"spinlock_wait_cycles":19200,"thread_start_us":5275,"threads_started":1}
I20260807 02:56:29.748461 11000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:29.749372 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.696580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.66:56145)
0807 02:56:29.720691 (+ 24111us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:29.720718 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:29.722625 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:29.723496 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:29.723507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:29.724906 (+ 1399us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:29.725894 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.725909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.727689 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.727694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.729574 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.729589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.730679 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.730691 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.731043 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.732391 (+ 1348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:29.732422 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:29.736439 (+ 4017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:29.741327 (+ 4888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.741336 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.741355 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.742172 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.742657 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.742661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.742664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.742993 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.743728 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:29.743737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:29.744269 (+ 532us) client_negotiation.cc:769] Sending connection context
0807 02:56:29.745787 (+ 1518us) client_negotiation.cc:240] Negotiation successful
0807 02:56:29.746307 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23948,"mutex_wait_us":277,"spinlock_wait_cycles":67328,"thread_start_us":24168,"threads_started":1}
I20260807 02:56:29.749539 11938 heartbeater.cc:347] Connected to a master server at 127.1.121.124:37441
I20260807 02:56:29.749955 11938 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:29.750279 11939 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41229
I20260807 02:56:29.750463 11939 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:29.751237 11938 heartbeater.cc:511] Master 127.1.121.124:37441 requested a full tablet report, sending...
I20260807 02:56:29.751317 11939 heartbeater.cc:511] Master 127.1.121.126:41229 requested a full tablet report, sending...
I20260807 02:56:29.753193 11472 ts_manager.cc:194] Registered new tserver with Master: b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947)
I20260807 02:56:29.753774 10356 ts_manager.cc:194] Re-registered known tserver with Master: b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947)
I20260807 02:56:29.754030 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:29.697124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.1.121.66:47817)
0807 02:56:29.721300 (+ 24176us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:29.721321 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:29.722625 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:29.723496 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:29.723507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:29.725903 (+ 2396us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:29.726214 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.726222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.728982 (+ 2760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.728989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.730723 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:29.730736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:29.732005 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:29.732012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:29.732162 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:29.732813 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:29.732879 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:29.736643 (+ 3764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:29.741570 (+ 4927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.741575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.741591 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.742371 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.742882 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:29.742887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:29.742890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:29.743119 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:29.743932 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:29.743937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:29.744274 (+ 337us) client_negotiation.cc:769] Sending connection context
0807 02:56:29.746314 (+ 2040us) client_negotiation.cc:240] Negotiation successful
0807 02:56:29.746598 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24012,"mutex_wait_us":24,"spinlock_wait_cycles":12032,"thread_start_us":23188,"threads_started":1}
I20260807 02:56:29.754825 10356 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:56145
I20260807 02:56:29.754881 11940 heartbeater.cc:347] Connected to a master server at 127.1.121.125:35089
I20260807 02:56:29.755020 11940 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:29.755715 11940 heartbeater.cc:511] Master 127.1.121.125:35089 requested a full tablet report, sending...
I20260807 02:56:29.756798 11207 ts_manager.cc:194] Registered new tserver with Master: b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947)
I20260807 02:56:29.839458 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 10830
I20260807 02:56:29.862938 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:34967
--local_ip_for_outbound_sockets=127.1.121.67
--tserver_master_addrs=127.1.121.126:41229
--webserver_port=38105
--webserver_interface=127.1.121.67
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab
--principal=oryx/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.121.126:41229,127.1.121.125:35089
--tserver_master_addrs=127.1.121.126:41229,127.1.121.125:35089,127.1.121.124:37441 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:56:30.091634 11725 heartbeater.cc:503] Master 127.1.121.126:41229 was elected leader, sending a full tablet report...
W20260807 02:56:30.231714 11957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:30.232106 11957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:30.232282 11957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:30.240154 11957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:30.240259 11957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:30.240378 11957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:56:30.250506 11957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab
--principal=oryx/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:34967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=38105
--tserver_master_addrs=127.1.121.126:41229,127.1.121.125:35089,127.1.121.124:37441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:30.251921 11957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:30.253328 11957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:30.268654 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:30.268780 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:30.268702 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:30.269569 11957 server_base.cc:1034] running on GCE node
I20260807 02:56:30.270192 11957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:30.271212 11957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:30.272444 11957 hybrid_clock.cc:648] HybridClock initialized: now 1786071390272372 us; error 67 us; skew 500 ppm
I20260807 02:56:30.272742 11957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:30 dist-test-slave-3h5h krb5kdc[10290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071390, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:30.280234 11957 init.cc:377] Logged in from keytab as oryx/127.1.121.67@KRBTEST.COM (short username oryx)
I20260807 02:56:30.281885 11957 webserver.cc:460] Webserver started at http://127.1.121.67:38105/ using document root <none> and password file <none>
I20260807 02:56:30.282649 11957 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:30.282817 11957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:30.288710 11957 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260807 02:56:30.291958 11981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:30.293524 11957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 02:56:30.293762 11957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "caedd08455f64c838304ade2db51fd36"
format_stamp: "Formatted at 2026-08-07 02:56:26 on dist-test-slave-3h5h"
server_key: "375ae107bbb1d0ed6f58835b69667854"
server_key_iv: "37bfcaacab25bdba6034b741ec8e40b5"
server_key_version: "encryptionkey@0"
I20260807 02:56:30.294788 11957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:30.311795 11957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:30.315980 11957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:30.316306 11957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:30.317523 11957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:30.319083 11957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:30.319201 11957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:30.319348 11957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:30.319473 11957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:30.357668 11957 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:34967
I20260807 02:56:30.357740 12132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:34967 every 8 connection(s)
I20260807 02:56:30.359205 11957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:56:30.361564 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 11957
Aug 07 02:56:30 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071390, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.125@KRBTEST.COM
Aug 07 02:56:30 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071390, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.124@KRBTEST.COM
Aug 07 02:56:30 dist-test-slave-3h5h krb5kdc[10290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071390, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:30.408028 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.362573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:43635 (local address 127.1.121.126:41229)
0807 02:56:30.375399 (+ 12826us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.375409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.383273 (+ 7864us) server_negotiation.cc:407] Connection header received
0807 02:56:30.384236 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.384242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.384336 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.384465 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.387239 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.388002 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.391217 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.391871 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.391988 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.399714 (+ 7726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.399722 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.399727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.399775 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.404822 (+ 5047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.405495 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.405504 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.405511 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.405725 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.406082 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.406091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.406097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.406553 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.406717 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.407102 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.407429 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12638,"thread_start_us":12834,"threads_started":1}
I20260807 02:56:30.410291 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.367843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:36775 (local address 127.1.121.124:37441)
0807 02:56:30.377257 (+ 9414us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.377262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.384878 (+ 7616us) server_negotiation.cc:407] Connection header received
0807 02:56:30.385716 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.385725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.385859 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.387759 (+ 1900us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.388016 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.389503 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.392324 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.393310 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.394361 (+ 1051us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.397677 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.397686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.397694 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.397747 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.402730 (+ 4983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.403806 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.403816 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.403822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.404016 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.404632 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.404640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.404646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.405162 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.405332 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.409054 (+ 3722us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.409812 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9296,"thread_start_us":9178,"threads_started":1}
I20260807 02:56:30.412045 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.363909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:46793 (local address 127.1.121.125:35089)
0807 02:56:30.385141 (+ 21232us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.385149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.385169 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:56:30.385233 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.385240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.385323 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.385435 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.388233 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.389348 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.391130 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.391809 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.391957 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.396185 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.396215 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.396221 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.396271 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.400260 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.401054 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.401060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.401063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.401236 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.409144 (+ 7908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.409150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.409154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.409585 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.410434 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.411379 (+ 945us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.411605 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21081,"thread_start_us":20817,"threads_started":1}
I20260807 02:56:30.412160 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.364281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.67:43635)
0807 02:56:30.383089 (+ 18808us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:30.383123 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:30.383939 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:30.384767 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:30.384779 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:30.385906 (+ 1127us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:30.387006 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.387033 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.388178 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.388183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.390513 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.390544 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.392943 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.392954 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.394786 (+ 1832us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.396232 (+ 1446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:30.396255 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:30.399463 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:30.404970 (+ 5507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:30.404977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:30.404981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:30.405355 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:30.405826 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:30.405832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:30.405836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:30.405959 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:30.406682 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:30.406687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:30.406840 (+ 153us) client_negotiation.cc:769] Sending connection context
0807 02:56:30.406976 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 02:56:30.410708 (+ 3732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18623,"mutex_wait_us":1778,"thread_start_us":16859,"threads_started":1}
I20260807 02:56:30.412648 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.367922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:37441 (local address 127.1.121.67:36775)
0807 02:56:30.383796 (+ 15874us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:30.383814 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:30.383942 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:30.386098 (+ 2156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:30.386104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:30.387022 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:30.387446 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.387463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.389763 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.389776 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.392056 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.392078 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.393587 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.393596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.393924 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.394642 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:30.394663 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:30.397369 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:30.402884 (+ 5515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:30.402892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:30.402899 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:30.403593 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:30.404181 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:30.404189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:30.404196 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:30.404457 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:30.405442 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:30.405454 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:30.406061 (+ 607us) client_negotiation.cc:769] Sending connection context
0807 02:56:30.406484 (+ 423us) client_negotiation.cc:240] Negotiation successful
0807 02:56:30.407276 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15750,"mutex_wait_us":1,"thread_start_us":15380,"threads_started":1}
I20260807 02:56:30.414800 12133 heartbeater.cc:347] Connected to a master server at 127.1.121.124:37441
I20260807 02:56:30.413142 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.363171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.1.121.67:46793)
0807 02:56:30.381521 (+ 18350us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:30.381584 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:30.383870 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:30.385708 (+ 1838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:30.385712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:30.386203 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:30.387006 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.387033 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.389529 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.389534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.390984 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.390995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.392319 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.392325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.392524 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.393276 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:30.393300 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:30.395953 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:30.400395 (+ 4442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:30.400404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:30.400422 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:30.400916 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:30.402935 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:30.402943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:30.402949 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:30.403253 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:30.410397 (+ 7144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:30.410405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:30.411100 (+ 695us) client_negotiation.cc:769] Sending connection context
0807 02:56:30.411280 (+ 180us) client_negotiation.cc:240] Negotiation successful
0807 02:56:30.411605 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18019,"mutex_wait_us":80,"thread_start_us":20172,"threads_started":1}
I20260807 02:56:30.415187 12133 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:30.416149 12133 heartbeater.cc:511] Master 127.1.121.124:37441 requested a full tablet report, sending...
I20260807 02:56:30.416918 12134 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41229
I20260807 02:56:30.417070 12134 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:30.417587 12134 heartbeater.cc:511] Master 127.1.121.126:41229 requested a full tablet report, sending...
I20260807 02:56:30.419178 12135 heartbeater.cc:347] Connected to a master server at 127.1.121.125:35089
I20260807 02:56:30.419328 12135 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:30.419976 12135 heartbeater.cc:511] Master 127.1.121.125:35089 requested a full tablet report, sending...
I20260807 02:56:30.420066 11472 ts_manager.cc:194] Registered new tserver with Master: caedd08455f64c838304ade2db51fd36 (127.1.121.67:34967)
I20260807 02:56:30.421149 10356 ts_manager.cc:194] Re-registered known tserver with Master: caedd08455f64c838304ade2db51fd36 (127.1.121.67:34967)
I20260807 02:56:30.421838 11207 ts_manager.cc:194] Registered new tserver with Master: caedd08455f64c838304ade2db51fd36 (127.1.121.67:34967)
I20260807 02:56:30.422487 10356 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:43635
I20260807 02:56:30.440737 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.402526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.1.121.126:41229)
0807 02:56:30.408767 (+ 6241us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.408776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.412112 (+ 3336us) server_negotiation.cc:407] Connection header received
0807 02:56:30.415668 (+ 3556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.415678 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.415790 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.415922 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.423511 (+ 7589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.425063 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.428310 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.429218 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.429368 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.433368 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.433376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.433384 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.433428 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.437227 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.438467 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.438474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.438478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.438613 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.439023 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.439027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.439029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.439463 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.439750 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.440174 (+ 424us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.440418 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6096,"thread_start_us":26220,"threads_started":1}
I20260807 02:56:30.444531 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.408513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.1.121.124:37441)
0807 02:56:30.409375 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.409380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.422895 (+ 13515us) server_negotiation.cc:407] Connection header received
0807 02:56:30.425057 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.425066 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.425174 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.425307 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.429447 (+ 4140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.430485 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.432038 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.432601 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.432732 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.437000 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.437008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.437015 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.437059 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.441381 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.442229 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.442234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.442237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.442427 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.442936 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.442942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.442948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.443293 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.443577 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.443919 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.444153 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":452,"threads_started":1}
I20260807 02:56:30.447101 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.409481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.1.121.125:35089)
0807 02:56:30.410177 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.410186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.428215 (+ 18029us) server_negotiation.cc:407] Connection header received
0807 02:56:30.428634 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.428640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.428732 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.428876 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.431943 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.433317 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.434921 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.435637 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.435778 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.439292 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.439297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.439300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.439363 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.443448 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.444612 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.444617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.444620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.444817 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.445267 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.445272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.445275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.445655 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.445947 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.446445 (+ 498us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.446691 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":358,"threads_started":1}
I20260807 02:56:30.445649 10356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:56:30.453720 10356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:56:30.512374 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.477096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.1.121.66:35947)
0807 02:56:30.504349 (+ 27253us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.504361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.504652 (+ 291us) server_negotiation.cc:407] Connection header received
0807 02:56:30.504793 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.504804 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.505140 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.505387 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:30.506439 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.507751 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.509979 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.511289 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.511456 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.511704 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.511925 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.512088 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27026,"thread_start_us":27270,"threads_started":1}
I20260807 02:56:30.512404 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.476773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:35947 (local address 127.0.0.1:58594)
0807 02:56:30.504332 (+ 27559us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:30.504352 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:30.504473 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:30.505392 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:30.505396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:30.505423 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:30.505790 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.505804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.507898 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.507906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.509799 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.509825 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.511513 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.511519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.511699 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.511743 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 02:56:30.511870 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 02:56:30.512008 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27438,"thread_start_us":27256,"threads_started":1}
I20260807 02:56:30.514041 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.476033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:45317 (local address 127.0.0.1:39352)
0807 02:56:30.503385 (+ 27352us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:30.503414 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:30.503598 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:30.506221 (+ 2623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:30.506225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:30.506249 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:30.506998 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.507011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.508947 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.508953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.511299 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.511315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.513334 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.513342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.513512 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.513547 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 02:56:30.513647 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 02:56:30.513763 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27123,"mutex_wait_us":1,"thread_start_us":28502,"threads_started":1}
I20260807 02:56:30.514835 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.476189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.1.121.65:45317)
0807 02:56:30.505546 (+ 29357us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.505555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.505596 (+ 41us) server_negotiation.cc:407] Connection header received
0807 02:56:30.505701 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.505709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.505989 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.506221 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:30.507198 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.508766 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.511484 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.513141 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.513277 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.513478 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.514330 (+ 852us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.514519 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29147,"thread_start_us":29384,"threads_started":1}
I20260807 02:56:30.515362 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.476619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:34967 (local address 127.0.0.1:53424)
0807 02:56:30.503645 (+ 27026us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:30.503668 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:30.503809 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:30.505939 (+ 2130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:30.505944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:30.505972 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:30.506521 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.506539 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.509525 (+ 2986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.509531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.511938 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.511956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.514490 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.514497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.514684 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.514720 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 02:56:30.514822 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 02:56:30.515140 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26865,"thread_start_us":25892,"threads_started":1}
I20260807 02:56:30.515445 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.477062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.1.121.67:34967)
0807 02:56:30.505266 (+ 28204us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.505276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.505314 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:56:30.505410 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.505416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.505636 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.505918 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:30.506896 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.509333 (+ 2437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.512147 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.514134 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.514422 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.514621 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.514886 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.515054 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27986,"thread_start_us":27791,"threads_started":1}
I20260807 02:56:30.517138 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 674a5eb715264e278890b06af8590c69 (DEFAULT_TABLE table=test-table [id=0317df8cd30e432b93cb357a5f944666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:30.518716 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674a5eb715264e278890b06af8590c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:30.519327 12050 tablet_service.cc:1467] Processing CreateTablet for tablet 674a5eb715264e278890b06af8590c69 (DEFAULT_TABLE table=test-table [id=0317df8cd30e432b93cb357a5f944666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:30.520634 12050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674a5eb715264e278890b06af8590c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:30.528357 11625 tablet_service.cc:1467] Processing CreateTablet for tablet 674a5eb715264e278890b06af8590c69 (DEFAULT_TABLE table=test-table [id=0317df8cd30e432b93cb357a5f944666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:30.530386 11625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674a5eb715264e278890b06af8590c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:30.532171 12181 tablet_bootstrap.cc:492] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: Bootstrap starting.
I20260807 02:56:30.535825 12182 tablet_bootstrap.cc:492] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: Bootstrap starting.
I20260807 02:56:30.536499 12181 tablet_bootstrap.cc:654] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:30.537968 12181 log.cc:824] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:30.539630 12182 tablet_bootstrap.cc:654] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:30.540953 12182 log.cc:824] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:30.541393 12181 tablet_bootstrap.cc:492] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: No bootstrap required, opened a new log
I20260807 02:56:30.541738 12181 ts_tablet_manager.cc:1397] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 02:56:30.545620 12186 tablet_bootstrap.cc:492] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: Bootstrap starting.
I20260807 02:56:30.549103 12186 tablet_bootstrap.cc:654] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:30.549470 12182 tablet_bootstrap.cc:492] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: No bootstrap required, opened a new log
I20260807 02:56:30.549762 12182 ts_tablet_manager.cc:1397] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260807 02:56:30.550355 12186 log.cc:824] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:30.553138 12186 tablet_bootstrap.cc:492] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: No bootstrap required, opened a new log
I20260807 02:56:30.553418 12186 ts_tablet_manager.cc:1397] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260807 02:56:30.553876 12181 raft_consensus.cc:348] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } }
I20260807 02:56:30.554436 12181 raft_consensus.cc:374] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:30.554559 12181 raft_consensus.cc:729] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64e2dc08e694040a7355cea1928fe9b, State: Initialized, Role: FOLLOWER
I20260807 02:56:30.555214 12181 consensus_queue.cc:260] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } }
I20260807 02:56:30.556998 12181 ts_tablet_manager.cc:1428] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260807 02:56:30.557289 11939 heartbeater.cc:503] Master 127.1.121.126:41229 was elected leader, sending a full tablet report...
I20260807 02:56:30.562976 12182 raft_consensus.cc:348] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } }
I20260807 02:56:30.563459 12182 raft_consensus.cc:374] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:30.563571 12182 raft_consensus.cc:729] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caedd08455f64c838304ade2db51fd36, State: Initialized, Role: FOLLOWER
I20260807 02:56:30.564273 12182 consensus_queue.cc:260] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } }
I20260807 02:56:30.566586 12182 ts_tablet_manager.cc:1428] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
I20260807 02:56:30.566749 12134 heartbeater.cc:503] Master 127.1.121.126:41229 was elected leader, sending a full tablet report...
I20260807 02:56:30.567281 12186 raft_consensus.cc:348] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } }
I20260807 02:56:30.567616 12190 raft_consensus.cc:482] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:30.567706 12186 raft_consensus.cc:374] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:30.567822 12186 raft_consensus.cc:729] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313d60cfddb645daaf6e649f261168c5, State: Initialized, Role: FOLLOWER
I20260807 02:56:30.567909 12190 raft_consensus.cc:504] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } }
I20260807 02:56:30.568563 12186 consensus_queue.cc:260] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } }
I20260807 02:56:30.574111 12190 leader_election.cc:288] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317), b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947)
I20260807 02:56:30.575932 12186 ts_tablet_manager.cc:1428] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: Time spent starting tablet: real 0.022s user 0.013s sys 0.006s
I20260807 02:56:30.580509 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.573009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:35947 (local address 127.1.121.67:59819)
0807 02:56:30.573257 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:30.573277 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:30.573366 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:30.573915 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:30.573920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:30.573946 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:30.574209 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.574227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.575812 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.575818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.577821 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.577836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.579653 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.579659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.579934 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.579970 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 02:56:30.580072 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 02:56:30.580217 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":27}
I20260807 02:56:30.581207 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.573198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:59819 (local address 127.1.121.66:35947)
0807 02:56:30.573636 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.573640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.573656 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:56:30.573726 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.573730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.573782 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.573895 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:30.574423 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.575546 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.578218 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.579487 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.579668 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.579747 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.580775 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.580942 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260807 02:56:30.582866 11875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674a5eb715264e278890b06af8590c69" candidate_uuid: "caedd08455f64c838304ade2db51fd36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64e2dc08e694040a7355cea1928fe9b" is_pre_election: true
I20260807 02:56:30.583420 11875 raft_consensus.cc:2454] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caedd08455f64c838304ade2db51fd36 in term 0.
I20260807 02:56:30.584348 11982 leader_election.cc:302] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b64e2dc08e694040a7355cea1928fe9b, caedd08455f64c838304ade2db51fd36; no voters:
I20260807 02:56:30.584808 12190 raft_consensus.cc:2781] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:56:30.585088 12190 raft_consensus.cc:482] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:56:30.585204 12190 raft_consensus.cc:3037] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:30.585392 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.571813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:40615 (local address 127.1.121.65:45317)
0807 02:56:30.572053 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.572059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.572078 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:56:30.572149 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.572155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.572235 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.572338 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:30.573416 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.574944 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.580152 (+ 5208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.584264 (+ 4112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.584431 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.584513 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.584978 (+ 465us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.585156 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:56:30.585639 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.571230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:45317 (local address 127.1.121.67:40615)
0807 02:56:30.571484 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:30.571512 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:30.571698 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:30.572384 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:30.572391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:30.572438 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:30.572761 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.572777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.575149 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.575154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.579990 (+ 4836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:30.580003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.584416 (+ 4413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.584423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:30.584733 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.584776 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 02:56:30.584955 (+ 179us) client_negotiation.cc:240] Negotiation successful
0807 02:56:30.585146 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":2625}
I20260807 02:56:30.586702 11654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674a5eb715264e278890b06af8590c69" candidate_uuid: "caedd08455f64c838304ade2db51fd36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313d60cfddb645daaf6e649f261168c5" is_pre_election: true
I20260807 02:56:30.587190 11654 raft_consensus.cc:2454] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caedd08455f64c838304ade2db51fd36 in term 0.
I20260807 02:56:30.588207 12190 raft_consensus.cc:504] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } }
I20260807 02:56:30.588804 12190 leader_election.cc:288] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [CANDIDATE]: Term 1 election: Requested vote from peers 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317), b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947)
I20260807 02:56:30.589232 11654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674a5eb715264e278890b06af8590c69" candidate_uuid: "caedd08455f64c838304ade2db51fd36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313d60cfddb645daaf6e649f261168c5"
I20260807 02:56:30.589416 11654 raft_consensus.cc:3037] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:30.589440 11875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674a5eb715264e278890b06af8590c69" candidate_uuid: "caedd08455f64c838304ade2db51fd36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64e2dc08e694040a7355cea1928fe9b"
I20260807 02:56:30.589625 11875 raft_consensus.cc:3037] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:30.592353 11875 raft_consensus.cc:2454] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caedd08455f64c838304ade2db51fd36 in term 1.
I20260807 02:56:30.592731 11654 raft_consensus.cc:2454] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caedd08455f64c838304ade2db51fd36 in term 1.
I20260807 02:56:30.592872 11982 leader_election.cc:302] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b64e2dc08e694040a7355cea1928fe9b, caedd08455f64c838304ade2db51fd36; no voters:
I20260807 02:56:30.593245 12190 raft_consensus.cc:2781] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:30.593904 12190 raft_consensus.cc:686] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 1 LEADER]: Becoming Leader. State: Replica: caedd08455f64c838304ade2db51fd36, State: Running, Role: LEADER
I20260807 02:56:30.594511 12190 consensus_queue.cc:237] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } }
I20260807 02:56:30.599615 10355 catalog_manager.cc:5495] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 reported cstate change: term changed from 0 to 1, leader changed from <none> to caedd08455f64c838304ade2db51fd36 (127.1.121.67). New cstate: current_term: 1 leader_uuid: "caedd08455f64c838304ade2db51fd36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } health_report { overall_health: UNKNOWN } } }
W20260807 02:56:30.617825 12136 tablet.cc:2369] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:56:30.620153 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.614333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.1.121.67:34967)
0807 02:56:30.614586 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.614591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.614610 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:56:30.614697 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.614701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.614787 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.614910 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:56:30.615239 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.615996 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.617464 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.618244 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.618376 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.618878 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:56:30.619488 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:56:30.619636 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.619783 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.619852 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 02:56:30.629349 11654 raft_consensus.cc:1261] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [term 1 FOLLOWER]: Refusing update from remote peer caedd08455f64c838304ade2db51fd36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:30.629423 11875 raft_consensus.cc:1261] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [term 1 FOLLOWER]: Refusing update from remote peer caedd08455f64c838304ade2db51fd36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:30.630839 12192 consensus_queue.cc:1035] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:30.631258 12190 consensus_queue.cc:1035] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:30.644194 12199 mvcc.cc:205] Tried to move back new op lower bound from 7315748416005263360 to 7315748415879655424. Current Snapshot: MvccSnapshot[applied={T|T < 7315748416005263360}]
I20260807 02:56:30.680634 10355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:30.680976 10355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:30.688215 10355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfcbe99e52f743b7ab27a4fba0ccfec9: Sending DeleteTablet for 3 replicas of tablet 674a5eb715264e278890b06af8590c69
I20260807 02:56:30.689486 11625 tablet_service.cc:1514] Processing DeleteTablet for tablet 674a5eb715264e278890b06af8590c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:30 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:39352
I20260807 02:56:30.689831 11847 tablet_service.cc:1514] Processing DeleteTablet for tablet 674a5eb715264e278890b06af8590c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:30 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:58594
I20260807 02:56:30.697887 12050 tablet_service.cc:1514] Processing DeleteTablet for tablet 674a5eb715264e278890b06af8590c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:30 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:53424
I20260807 02:56:30.703262 12210 tablet_replica.cc:330] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: stopping tablet replica
I20260807 02:56:30.704043 12210 raft_consensus.cc:2229] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:30.704866 12210 raft_consensus.cc:2258] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:30.707511 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 10311
I20260807 02:56:30.708767 10401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:30.717067 12210 ts_tablet_manager.cc:1905] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:30.726414 12213 tablet_replica.cc:330] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: stopping tablet replica
I20260807 02:56:30.727022 12213 raft_consensus.cc:2229] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:56:30.727772 12213 raft_consensus.cc:2258] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:30.729765 12210 ts_tablet_manager.cc:1918] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:30.734208 12210 log.cc:1196] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/674a5eb715264e278890b06af8590c69
I20260807 02:56:30.734449 12213 ts_tablet_manager.cc:1905] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:30.735101 12210 ts_tablet_manager.cc:1939] T 674a5eb715264e278890b06af8590c69 P 313d60cfddb645daaf6e649f261168c5: Deleting consensus metadata
I20260807 02:56:30.737741 12211 tablet_replica.cc:330] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: stopping tablet replica
I20260807 02:56:30.738260 12211 raft_consensus.cc:2229] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:30.743054 12211 raft_consensus.cc:2258] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:30.746049 12211 ts_tablet_manager.cc:1905] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:30.747457 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.692245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.1.121.125:35089)
0807 02:56:30.697458 (+ 5213us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.697468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.706131 (+ 8663us) server_negotiation.cc:407] Connection header received
0807 02:56:30.714649 (+ 8518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.714657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.714749 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.714822 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.722783 (+ 7961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.724116 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.731375 (+ 7259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.732281 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.732402 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.738945 (+ 6543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.739115 (+ 170us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.739125 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.739182 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.743083 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.744283 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.744289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.744292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.744442 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.745228 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.745234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.745237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.745745 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.746005 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.746828 (+ 823us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.747077 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4972}
I20260807 02:56:30.766079 12213 ts_tablet_manager.cc:1918] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:30.766297 12213 log.cc:1196] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/674a5eb715264e278890b06af8590c69
I20260807 02:56:30.767028 12213 ts_tablet_manager.cc:1939] T 674a5eb715264e278890b06af8590c69 P caedd08455f64c838304ade2db51fd36: Deleting consensus metadata
I20260807 02:56:30.770223 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.706239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.1.121.124:37441)
0807 02:56:30.706439 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.706446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.750231 (+ 43785us) server_negotiation.cc:407] Connection header received
0807 02:56:30.750371 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.750378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.750485 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.750603 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.752450 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.753790 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.756204 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.756966 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.757101 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.760606 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.760625 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.760632 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.760742 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.764715 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.767724 (+ 3009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.767730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.767734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.767916 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.768348 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.768355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.768360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.768783 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.769092 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.769601 (+ 509us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.769862 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:56:30.774345 12211 ts_tablet_manager.cc:1918] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:30.774582 12211 log.cc:1196] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/674a5eb715264e278890b06af8590c69
I20260807 02:56:30.775262 12211 ts_tablet_manager.cc:1939] T 674a5eb715264e278890b06af8590c69 P b64e2dc08e694040a7355cea1928fe9b: Deleting consensus metadata
I20260807 02:56:30.882944 10332 catalog_manager.cc:4841] TS b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947): tablet 674a5eb715264e278890b06af8590c69 (table test-table [id=0317df8cd30e432b93cb357a5f944666]) successfully deleted
I20260807 02:56:30.883390 10334 catalog_manager.cc:4841] TS caedd08455f64c838304ade2db51fd36 (127.1.121.67:34967): tablet 674a5eb715264e278890b06af8590c69 (table test-table [id=0317df8cd30e432b93cb357a5f944666]) successfully deleted
I20260807 02:56:30.883350 10333 catalog_manager.cc:4841] TS 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317): tablet 674a5eb715264e278890b06af8590c69 (table test-table [id=0317df8cd30e432b93cb357a5f944666]) successfully deleted
I20260807 02:56:30.883831 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 10311
I20260807 02:56:30.946269 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.919829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.1.121.125:35089)
0807 02:56:30.920060 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.920068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.928966 (+ 8898us) server_negotiation.cc:407] Connection header received
0807 02:56:30.929222 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.929230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.929330 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.929434 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.931718 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.932848 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.935025 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.935552 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.935656 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.939943 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.939949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.939958 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.939995 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.943344 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.944040 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.944045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.944051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.944178 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.944652 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.944659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.944663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.945061 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.945292 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.945700 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.945927 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 02:56:30.947460 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:30.919840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.1.121.124:37441)
0807 02:56:30.920065 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:30.920071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:30.929549 (+ 9478us) server_negotiation.cc:407] Connection header received
0807 02:56:30.929697 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:30.929703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:30.929793 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:30.929904 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:30.932283 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.933344 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.935144 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:30.935795 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:30.935935 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:30.939910 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:30.939917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:30.939923 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:30.939983 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:30.944149 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.945033 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.945038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.945042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.945223 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:30.945673 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:30.945678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:30.945681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:30.946084 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:30.946296 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:30.946888 (+ 592us) server_negotiation.cc:299] Negotiation successful
0807 02:56:30.947130 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 02:56:31.651924 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:31.651042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.66:57939)
0807 02:56:31.651623 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:31.651750 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":311,"threads_started":1}
W20260807 02:56:31.652472 11939 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:41229 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
I20260807 02:56:31.664044 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:31.662967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.67:33445)
0807 02:56:31.663583 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:31.663766 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":266,"threads_started":1}
W20260807 02:56:31.664575 12134 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:41229 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
I20260807 02:56:31.673974 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:31.673106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.65:52929)
0807 02:56:31.673647 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:31.673775 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":325,"threads_started":1}
W20260807 02:56:31.674412 11725 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:41229 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
I20260807 02:56:32.188910 12234 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfcbe99e52f743b7ab27a4fba0ccfec9)
I20260807 02:56:32.189205 12234 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } }
I20260807 02:56:32.190611 12234 leader_election.cc:288] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfcbe99e52f743b7ab27a4fba0ccfec9 (127.1.121.126:41229), ecd338538e82459a99489e04e9660fbf (127.1.121.124:37441)
I20260807 02:56:32.197902 12239 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfcbe99e52f743b7ab27a4fba0ccfec9)
I20260807 02:56:32.198160 12239 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } }
I20260807 02:56:32.198845 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.190754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41906)
0807 02:56:32.198129 (+ 7375us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.198427 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7241,"thread_start_us":7096,"threads_started":1}
W20260807 02:56:32.199190 11187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
I20260807 02:56:32.199443 12239 leader_election.cc:288] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfcbe99e52f743b7ab27a4fba0ccfec9 (127.1.121.126:41229), e4ad78b04e534f29b0d7fd563efb533c (127.1.121.125:35089)
I20260807 02:56:32.210079 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.190945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:37441 (local address 127.0.0.1:45934)
0807 02:56:32.199074 (+ 8129us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.199092 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:32.199268 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:32.200031 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:32.200039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:32.200096 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:32.201032 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.201056 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.203197 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.203205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.206692 (+ 3487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.206716 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.209225 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.209232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.209421 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.209460 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 02:56:32.209547 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 02:56:32.209702 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8020,"thread_start_us":7775,"threads_started":1}
I20260807 02:56:32.210258 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.209769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41922)
0807 02:56:32.210018 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.210131 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260807 02:56:32.210458 11187 leader_election.cc:334] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfcbe99e52f743b7ab27a4fba0ccfec9 (127.1.121.126:41229): Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
I20260807 02:56:32.210875 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.191102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.1.121.124:37441)
0807 02:56:32.198563 (+ 7461us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:32.198570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:32.199197 (+ 627us) server_negotiation.cc:407] Connection header received
0807 02:56:32.199506 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:32.199512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:32.199594 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:32.199682 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:32.201295 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.202920 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.206925 (+ 4005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.208768 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.208922 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.209051 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:32.210326 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0807 02:56:32.210479 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7326,"thread_start_us":7191,"threads_started":1}
I20260807 02:56:32.211753 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.199497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41910)
0807 02:56:32.211128 (+ 11631us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.211375 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11491,"thread_start_us":11587,"threads_started":1}
W20260807 02:56:32.212111 11450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
I20260807 02:56:32.212505 11485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4ad78b04e534f29b0d7fd563efb533c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ecd338538e82459a99489e04e9660fbf" is_pre_election: true
I20260807 02:56:32.213014 11485 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ad78b04e534f29b0d7fd563efb533c in term 1.
I20260807 02:56:32.213645 11190 leader_election.cc:302] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e4ad78b04e534f29b0d7fd563efb533c, ecd338538e82459a99489e04e9660fbf; no voters: dfcbe99e52f743b7ab27a4fba0ccfec9
I20260807 02:56:32.214145 12234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 02:56:32.214288 12234 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfcbe99e52f743b7ab27a4fba0ccfec9)
I20260807 02:56:32.214411 12234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:56:32.214433 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.213850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41928)
0807 02:56:32.214086 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.214180 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260807 02:56:32.214641 11450 leader_election.cc:334] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfcbe99e52f743b7ab27a4fba0ccfec9 (127.1.121.126:41229): Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
I20260807 02:56:32.217736 12234 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } }
I20260807 02:56:32.218545 12234 leader_election.cc:288] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [CANDIDATE]: Term 2 election: Requested vote from peers dfcbe99e52f743b7ab27a4fba0ccfec9 (127.1.121.126:41229), ecd338538e82459a99489e04e9660fbf (127.1.121.124:37441)
I20260807 02:56:32.219074 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.218614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41930)
0807 02:56:32.218782 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.218910 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 02:56:32.219269 11485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4ad78b04e534f29b0d7fd563efb533c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ecd338538e82459a99489e04e9660fbf"
I20260807 02:56:32.219391 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.200010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:35089 (local address 127.0.0.1:45874)
0807 02:56:32.211930 (+ 11920us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.211950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:32.212111 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:32.212536 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:32.212543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:32.212571 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:32.213043 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.213068 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.214960 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.214965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.217017 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.217033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.218463 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.218470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.218709 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.218743 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 02:56:32.218868 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 02:56:32.219066 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11791,"thread_start_us":11594,"threads_started":1}
I20260807 02:56:32.219512 11485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:56:32.220439 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.220112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41936)
0807 02:56:32.220263 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.220337 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 02:56:32.220682 11187 leader_election.cc:334] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dfcbe99e52f743b7ab27a4fba0ccfec9 (127.1.121.126:41229): Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
I20260807 02:56:32.221820 11485 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ad78b04e534f29b0d7fd563efb533c in term 2.
I20260807 02:56:32.222239 11190 leader_election.cc:302] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e4ad78b04e534f29b0d7fd563efb533c, ecd338538e82459a99489e04e9660fbf; no voters: dfcbe99e52f743b7ab27a4fba0ccfec9
I20260807 02:56:32.222509 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.205645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.1.121.125:35089)
0807 02:56:32.211470 (+ 5825us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:32.211476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:32.212045 (+ 569us) server_negotiation.cc:407] Connection header received
0807 02:56:32.212275 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:32.212279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:32.212374 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:32.212480 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:32.213340 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.214767 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.217209 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.218273 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.218411 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.218558 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:32.218908 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 02:56:32.222121 (+ 3213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5673,"thread_start_us":5619,"threads_started":1}
I20260807 02:56:32.222584 12234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 2 FOLLOWER]: Leader election won for term 2
I20260807 02:56:32.223071 11219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecd338538e82459a99489e04e9660fbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4ad78b04e534f29b0d7fd563efb533c" is_pre_election: true
I20260807 02:56:32.223354 12234 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [term 2 LEADER]: Becoming Leader. State: Replica: e4ad78b04e534f29b0d7fd563efb533c, State: Running, Role: LEADER
I20260807 02:56:32.223765 12234 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } }
I20260807 02:56:32.226539 12247 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4ad78b04e534f29b0d7fd563efb533c. Latest consensus state: current_term: 2 leader_uuid: "e4ad78b04e534f29b0d7fd563efb533c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:32.227109 12247 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:32.227394 11453 leader_election.cc:302] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecd338538e82459a99489e04e9660fbf; no voters: dfcbe99e52f743b7ab27a4fba0ccfec9, e4ad78b04e534f29b0d7fd563efb533c
I20260807 02:56:32.227824 12239 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:56:32.227978 12249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:32.233088 12249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:32.234041 12249 catalog_manager.cc:1259] Loaded cluster ID: b7c877b6dc8a48c1b68079f078964abb
I20260807 02:56:32.234136 12249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:32.235378 12249 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:32.236200 12249 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Loaded TSK: 0
I20260807 02:56:32.237185 12249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:56:32.291787 11207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:56:32.300896 11207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:56:32.306025 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.305510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41950)
0807 02:56:32.305724 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.305853 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 02:56:32.306119 11485 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [term 2 FOLLOWER]: Refusing update from remote peer e4ad78b04e534f29b0d7fd563efb533c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260807 02:56:32.306915 12247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260807 02:56:32.307260 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.306787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41960)
0807 02:56:32.306987 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.307064 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 02:56:32.307492 11187 consensus_peers.cc:487] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c -> Peer dfcbe99e52f743b7ab27a4fba0ccfec9 (127.1.121.126:41229): Couldn't send request to peer dfcbe99e52f743b7ab27a4fba0ccfec9. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 02:56:32.310077 12239 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4ad78b04e534f29b0d7fd563efb533c. Latest consensus state: current_term: 2 leader_uuid: "e4ad78b04e534f29b0d7fd563efb533c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:32.310389 12239 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:56:32.311841 12234 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e4ad78b04e534f29b0d7fd563efb533c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:32.312251 12234 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:32.312379 12239 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e4ad78b04e534f29b0d7fd563efb533c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "dfcbe99e52f743b7ab27a4fba0ccfec9" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41229 } } peers { permanent_uuid: "e4ad78b04e534f29b0d7fd563efb533c" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 35089 } attrs { promote: false } } peers { permanent_uuid: "ecd338538e82459a99489e04e9660fbf" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 37441 } attrs { promote: false } } }
I20260807 02:56:32.312664 12239 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd338538e82459a99489e04e9660fbf [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:56:32.328562 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.320334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:35947 (local address 127.0.0.1:58598)
0807 02:56:32.320887 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.320917 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:32.321048 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:32.321452 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:32.321457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:32.321484 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:32.321851 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.321868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.323435 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.323443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.325827 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.325842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.327715 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.327724 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.327920 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.327970 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 02:56:32.328047 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 02:56:32.328187 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374}
I20260807 02:56:32.329962 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.320522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.1.121.66:35947)
0807 02:56:32.321104 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:32.321113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:32.321135 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:56:32.321211 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:32.321218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:32.321302 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:32.321421 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:32.322061 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.323267 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.326067 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.327413 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.329273 (+ 1860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.329394 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:32.329455 (+ 61us) server_negotiation.cc:299] Negotiation successful
0807 02:56:32.329625 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":616,"threads_started":1}
I20260807 02:56:32.330763 11847 tablet_service.cc:1467] Processing CreateTablet for tablet a7649b64d1874198a445f2702bc648a8 (DEFAULT_TABLE table=test-table [id=2f001276e6e24c1ba8da388e4604f830]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:32.331334 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7649b64d1874198a445f2702bc648a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:32.332059 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.320825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:45317 (local address 127.0.0.1:39358)
0807 02:56:32.321206 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.321222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:32.321319 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:32.323616 (+ 2297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:32.323624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:32.323664 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:32.323963 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.323977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.326051 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.326056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.328115 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.328128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.331063 (+ 2935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.331070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.331456 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.331493 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 02:56:32.331571 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 02:56:32.331716 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259}
I20260807 02:56:32.334236 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.321578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.1.121.65:45317)
0807 02:56:32.322114 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:32.322124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:32.322147 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:56:32.322222 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:32.322230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:32.322306 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:32.324137 (+ 1831us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:32.324329 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.325844 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.329523 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.330823 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.332370 (+ 1547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.333615 (+ 1245us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:32.333711 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 02:56:32.333884 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":520,"threads_started":1}
I20260807 02:56:32.335062 11625 tablet_service.cc:1467] Processing CreateTablet for tablet a7649b64d1874198a445f2702bc648a8 (DEFAULT_TABLE table=test-table [id=2f001276e6e24c1ba8da388e4604f830]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:32.335575 11625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7649b64d1874198a445f2702bc648a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:32.338837 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.321925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.1.121.67:34967)
0807 02:56:32.333140 (+ 11215us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:32.333145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:32.333162 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:56:32.333222 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:32.333225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:32.333281 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:32.333412 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:32.333828 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.335030 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.336732 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.337781 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.337927 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.338030 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:32.338324 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 02:56:32.338482 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11030,"thread_start_us":11606,"threads_started":1}
I20260807 02:56:32.338837 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.321228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:34967 (local address 127.0.0.1:53444)
0807 02:56:32.328831 (+ 7603us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.328873 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:32.328971 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:32.333427 (+ 4456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:32.333432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:32.333456 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:32.333657 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.333669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.335216 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.335221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.336563 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:32.336575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.337934 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.337940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:32.338166 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.338190 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:56:32.338296 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 02:56:32.338551 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7491,"thread_start_us":10724,"threads_started":1}
I20260807 02:56:32.339784 12050 tablet_service.cc:1467] Processing CreateTablet for tablet a7649b64d1874198a445f2702bc648a8 (DEFAULT_TABLE table=test-table [id=2f001276e6e24c1ba8da388e4604f830]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:32.340204 12050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7649b64d1874198a445f2702bc648a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:32.343458 12265 tablet_bootstrap.cc:492] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: Bootstrap starting.
I20260807 02:56:32.344596 12267 tablet_bootstrap.cc:492] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: Bootstrap starting.
I20260807 02:56:32.345484 12265 tablet_bootstrap.cc:654] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:32.346423 12268 tablet_bootstrap.cc:492] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: Bootstrap starting.
I20260807 02:56:32.346498 12267 tablet_bootstrap.cc:654] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:32.348723 12265 tablet_bootstrap.cc:492] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: No bootstrap required, opened a new log
I20260807 02:56:32.348917 12265 ts_tablet_manager.cc:1397] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:56:32.348955 12267 tablet_bootstrap.cc:492] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: No bootstrap required, opened a new log
I20260807 02:56:32.349118 12267 ts_tablet_manager.cc:1397] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 02:56:32.349359 12268 tablet_bootstrap.cc:654] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:32.349836 12267 raft_consensus.cc:348] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } }
I20260807 02:56:32.350049 12267 raft_consensus.cc:374] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:32.349937 12265 raft_consensus.cc:348] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } }
I20260807 02:56:32.350152 12267 raft_consensus.cc:729] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313d60cfddb645daaf6e649f261168c5, State: Initialized, Role: FOLLOWER
I20260807 02:56:32.350242 12265 raft_consensus.cc:374] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:32.350370 12265 raft_consensus.cc:729] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64e2dc08e694040a7355cea1928fe9b, State: Initialized, Role: FOLLOWER
I20260807 02:56:32.350407 12267 consensus_queue.cc:260] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } }
I20260807 02:56:32.350729 12265 consensus_queue.cc:260] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } }
I20260807 02:56:32.352175 12268 tablet_bootstrap.cc:492] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: No bootstrap required, opened a new log
I20260807 02:56:32.352418 12268 ts_tablet_manager.cc:1397] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:56:32.353652 12268 raft_consensus.cc:348] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } }
I20260807 02:56:32.354025 12268 raft_consensus.cc:374] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:32.354171 12268 raft_consensus.cc:729] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caedd08455f64c838304ade2db51fd36, State: Initialized, Role: FOLLOWER
I20260807 02:56:32.354493 12268 consensus_queue.cc:260] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } }
I20260807 02:56:32.355680 12268 ts_tablet_manager.cc:1428] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 02:56:32.359700 12267 ts_tablet_manager.cc:1428] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20260807 02:56:32.360471 11726 heartbeater.cc:503] Master 127.1.121.125:35089 was elected leader, sending a full tablet report...
I20260807 02:56:32.363511 12135 heartbeater.cc:503] Master 127.1.121.125:35089 was elected leader, sending a full tablet report...
I20260807 02:56:32.373131 12265 ts_tablet_manager.cc:1428] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: Time spent starting tablet: real 0.020s user 0.005s sys 0.005s
I20260807 02:56:32.373160 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.369703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.66:34489)
0807 02:56:32.370705 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.370924 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":823,"threads_started":1}
I20260807 02:56:32.374226 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.355634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.65:51515)
0807 02:56:32.373754 (+ 18120us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.373952 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17956,"thread_start_us":17759,"threads_started":1}
I20260807 02:56:32.374650 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.356400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.67:53173)
0807 02:56:32.374082 (+ 17682us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.374281 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17511,"thread_start_us":16661,"threads_started":1}
I20260807 02:56:32.375150 11940 heartbeater.cc:503] Master 127.1.121.125:35089 was elected leader, sending a full tablet report...
I20260807 02:56:32.375906 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.375452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.66:39011)
0807 02:56:32.375683 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.375768 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260807 02:56:32.376183 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.375320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.67:50629)
0807 02:56:32.375907 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.376014 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469}
I20260807 02:56:32.376889 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.375645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.65:41787)
0807 02:56:32.376587 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.376692 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":804}
W20260807 02:56:32.448227 11941 tablet.cc:2369] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:56:32.536535 12276 raft_consensus.cc:482] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:32.536801 12276 raft_consensus.cc:504] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } }
I20260807 02:56:32.537698 12276 leader_election.cc:288] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947), 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317)
I20260807 02:56:32.538371 11875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7649b64d1874198a445f2702bc648a8" candidate_uuid: "caedd08455f64c838304ade2db51fd36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64e2dc08e694040a7355cea1928fe9b" is_pre_election: true
I20260807 02:56:32.538372 11654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7649b64d1874198a445f2702bc648a8" candidate_uuid: "caedd08455f64c838304ade2db51fd36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313d60cfddb645daaf6e649f261168c5" is_pre_election: true
I20260807 02:56:32.538741 11875 raft_consensus.cc:2454] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caedd08455f64c838304ade2db51fd36 in term 0.
I20260807 02:56:32.538725 11654 raft_consensus.cc:2454] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caedd08455f64c838304ade2db51fd36 in term 0.
I20260807 02:56:32.539357 11983 leader_election.cc:302] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 313d60cfddb645daaf6e649f261168c5, b64e2dc08e694040a7355cea1928fe9b, caedd08455f64c838304ade2db51fd36; no voters:
I20260807 02:56:32.539800 12276 raft_consensus.cc:2781] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:56:32.539940 12276 raft_consensus.cc:482] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:56:32.540036 12276 raft_consensus.cc:3037] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:32.543385 12276 raft_consensus.cc:504] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } }
I20260807 02:56:32.543989 12276 leader_election.cc:288] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [CANDIDATE]: Term 1 election: Requested vote from peers b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947), 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317)
I20260807 02:56:32.544561 11875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7649b64d1874198a445f2702bc648a8" candidate_uuid: "caedd08455f64c838304ade2db51fd36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64e2dc08e694040a7355cea1928fe9b"
I20260807 02:56:32.544591 11654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7649b64d1874198a445f2702bc648a8" candidate_uuid: "caedd08455f64c838304ade2db51fd36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313d60cfddb645daaf6e649f261168c5"
I20260807 02:56:32.544797 11654 raft_consensus.cc:3037] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:32.544799 11875 raft_consensus.cc:3037] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:32.546878 11654 raft_consensus.cc:2454] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caedd08455f64c838304ade2db51fd36 in term 1.
I20260807 02:56:32.546896 11875 raft_consensus.cc:2454] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caedd08455f64c838304ade2db51fd36 in term 1.
I20260807 02:56:32.547327 11983 leader_election.cc:302] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 313d60cfddb645daaf6e649f261168c5, caedd08455f64c838304ade2db51fd36; no voters:
I20260807 02:56:32.547597 12276 raft_consensus.cc:2781] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:32.548055 12276 raft_consensus.cc:686] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 1 LEADER]: Becoming Leader. State: Replica: caedd08455f64c838304ade2db51fd36, State: Running, Role: LEADER
I20260807 02:56:32.548621 12276 consensus_queue.cc:237] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } }
I20260807 02:56:32.549297 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.548746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.67:52919)
0807 02:56:32.548980 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.549115 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260807 02:56:32.551991 11206 catalog_manager.cc:5495] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 reported cstate change: term changed from 0 to 1, leader changed from <none> to caedd08455f64c838304ade2db51fd36 (127.1.121.67). New cstate: current_term: 1 leader_uuid: "caedd08455f64c838304ade2db51fd36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caedd08455f64c838304ade2db51fd36" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34967 } health_report { overall_health: HEALTHY } } }
W20260807 02:56:32.553716 11727 tablet.cc:2369] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:56:32.576010 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.571420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.1.121.67:34967)
0807 02:56:32.571627 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:32.571632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:32.571650 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:56:32.571794 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:32.571798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:32.571861 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:32.571934 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:56:32.572308 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.573110 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.574414 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.574823 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.574913 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.575319 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:56:32.575416 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:56:32.575484 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:32.575637 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 02:56:32.575699 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:56:32.579542 11654 raft_consensus.cc:1261] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [term 1 FOLLOWER]: Refusing update from remote peer caedd08455f64c838304ade2db51fd36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:32.579787 11875 raft_consensus.cc:1261] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [term 1 FOLLOWER]: Refusing update from remote peer caedd08455f64c838304ade2db51fd36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:32.580421 12280 consensus_queue.cc:1035] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313d60cfddb645daaf6e649f261168c5" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:32.580770 12276 consensus_queue.cc:1035] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64e2dc08e694040a7355cea1928fe9b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 35947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:32.583528 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.582987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.65:35863)
0807 02:56:32.583220 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.583344 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260807 02:56:32.584684 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.584178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.66:49663)
0807 02:56:32.584367 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.584487 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 02:56:32.590430 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.589970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.66:60845)
0807 02:56:32.590176 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.590262 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260807 02:56:32.596627 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.594688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.67:52097)
0807 02:56:32.596289 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.596413 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1442}
I20260807 02:56:32.604624 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.603404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.67:37753)
0807 02:56:32.604271 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.604383 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":719}
I20260807 02:56:32.606956 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.606452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.1.121.65:58815)
0807 02:56:32.606683 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.606779 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260807 02:56:32.607398 11206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:32.607685 11206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:32.614326 11206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4ad78b04e534f29b0d7fd563efb533c: Sending DeleteTablet for 3 replicas of tablet a7649b64d1874198a445f2702bc648a8
I20260807 02:56:32.615502 11625 tablet_service.cc:1514] Processing DeleteTablet for tablet a7649b64d1874198a445f2702bc648a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:32 UTC) from {username='oryx', principal='oryx/127.1.121.125@KRBTEST.COM'} at 127.0.0.1:39358
I20260807 02:56:32.615558 11847 tablet_service.cc:1514] Processing DeleteTablet for tablet a7649b64d1874198a445f2702bc648a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:32 UTC) from {username='oryx', principal='oryx/127.1.121.125@KRBTEST.COM'} at 127.0.0.1:58598
I20260807 02:56:32.615896 11624 tablet_service.cc:1514] Processing DeleteTablet for tablet a7649b64d1874198a445f2702bc648a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:32 UTC) from {username='oryx', principal='oryx/127.1.121.125@KRBTEST.COM'} at 127.0.0.1:39358
I20260807 02:56:32.616099 12050 tablet_service.cc:1514] Processing DeleteTablet for tablet a7649b64d1874198a445f2702bc648a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:32 UTC) from {username='oryx', principal='oryx/127.1.121.125@KRBTEST.COM'} at 127.0.0.1:53444
I20260807 02:56:32.616490 12297 tablet_replica.cc:330] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: stopping tablet replica
I20260807 02:56:32.616801 12297 raft_consensus.cc:2229] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:32.617383 12297 raft_consensus.cc:2258] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:32.617456 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 11440
I20260807 02:56:32.622488 12297 ts_tablet_manager.cc:1905] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:32.622908 12300 tablet_replica.cc:330] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: stopping tablet replica
I20260807 02:56:32.623112 12300 raft_consensus.cc:2229] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:56:32.623196 12296 tablet_replica.cc:330] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: stopping tablet replica
I20260807 02:56:32.623422 12296 raft_consensus.cc:2229] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:32.623541 12300 raft_consensus.cc:2258] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:32.623821 12296 raft_consensus.cc:2258] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:32.624943 12300 ts_tablet_manager.cc:1905] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:32.627106 12296 ts_tablet_manager.cc:1905] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:32.629935 12297 ts_tablet_manager.cc:1918] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:32.630092 12297 log.cc:1196] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/a7649b64d1874198a445f2702bc648a8
I20260807 02:56:32.630555 12297 ts_tablet_manager.cc:1939] T a7649b64d1874198a445f2702bc648a8 P b64e2dc08e694040a7355cea1928fe9b: Deleting consensus metadata
I20260807 02:56:32.631827 11187 catalog_manager.cc:4841] TS b64e2dc08e694040a7355cea1928fe9b (127.1.121.66:35947): tablet a7649b64d1874198a445f2702bc648a8 (table test-table [id=2f001276e6e24c1ba8da388e4604f830]) successfully deleted
I20260807 02:56:32.632344 12300 ts_tablet_manager.cc:1918] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:32.632485 12300 log.cc:1196] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/a7649b64d1874198a445f2702bc648a8
I20260807 02:56:32.633021 12300 ts_tablet_manager.cc:1939] T a7649b64d1874198a445f2702bc648a8 P caedd08455f64c838304ade2db51fd36: Deleting consensus metadata
I20260807 02:56:32.633153 12296 ts_tablet_manager.cc:1918] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:32.633308 12296 log.cc:1196] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/a7649b64d1874198a445f2702bc648a8
I20260807 02:56:32.633810 12296 ts_tablet_manager.cc:1939] T a7649b64d1874198a445f2702bc648a8 P 313d60cfddb645daaf6e649f261168c5: Deleting consensus metadata
I20260807 02:56:32.634305 11189 catalog_manager.cc:4841] TS caedd08455f64c838304ade2db51fd36 (127.1.121.67:34967): tablet a7649b64d1874198a445f2702bc648a8 (table test-table [id=2f001276e6e24c1ba8da388e4604f830]) successfully deleted
I20260807 02:56:32.635119 11188 catalog_manager.cc:4841] TS 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317): tablet a7649b64d1874198a445f2702bc648a8 (table test-table [id=2f001276e6e24c1ba8da388e4604f830]) successfully deleted
W20260807 02:56:32.635592 11188 catalog_manager.cc:4808] TS 313d60cfddb645daaf6e649f261168c5 (127.1.121.65:45317): delete failed for tablet a7649b64d1874198a445f2702bc648a8 because the tablet was not found. No further retry: Not found: Tablet not found: a7649b64d1874198a445f2702bc648a8
I20260807 02:56:32.642774 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.618635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.1.121.65:45317)
0807 02:56:32.618915 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:32.618925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:32.624567 (+ 5642us) server_negotiation.cc:407] Connection header received
0807 02:56:32.624821 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:32.624827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:32.624943 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:32.625072 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:32.626533 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.627620 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.632083 (+ 4463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.632697 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.632909 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.636529 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:32.636549 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:32.636591 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:32.636622 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:32.639880 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:32.640586 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:32.640597 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:32.640600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:32.640735 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:32.641203 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:32.641209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:32.641212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:32.641605 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:32.641913 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:32.642221 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 02:56:32.642455 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 02:56:32.643510 11715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:32.756788 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 11440
I20260807 02:56:32.783946 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 11740
I20260807 02:56:32.797588 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.784819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.1.121.66:35947)
0807 02:56:32.785061 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:32.785075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:32.785096 (+ 21us) server_negotiation.cc:407] Connection header received
0807 02:56:32.785262 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:32.785266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:32.785333 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:32.785418 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:32.786745 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.787427 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.788605 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.789114 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.789208 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.791936 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:32.791952 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:32.791969 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:32.792009 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:32.794993 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:32.795610 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:32.795618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:32.795620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:32.795760 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:32.796117 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:32.796120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:32.796122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:32.796452 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:32.796695 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:32.796995 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 02:56:32.797184 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 02:56:32.798275 11933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:32.888594 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.887679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41968)
0807 02:56:32.888314 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.888411 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":392,"threads_started":1}
I20260807 02:56:32.891930 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.891492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41229 (local address 127.0.0.1:41970)
0807 02:56:32.891653 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:32.891747 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 02:56:32.902527 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 11740
I20260807 02:56:32.929697 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 11957
I20260807 02:56:32.944031 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:32.930496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.1.121.67:34967)
0807 02:56:32.930700 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:32.930705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:32.930725 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:56:32.930874 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:32.930878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:32.930945 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:32.931039 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:32.932587 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.933459 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.934846 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:32.935280 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:32.935423 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:32.938194 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:32.938212 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:32.938230 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:32.938281 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:32.941166 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:32.941852 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:32.941869 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:32.941871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:32.942024 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:32.942413 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:32.942418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:32.942421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:32.942794 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:32.943161 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:32.943464 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 02:56:32.943712 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 02:56:32.944744 12123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:33.070477 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 11957
I20260807 02:56:33.106055 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 11015
I20260807 02:56:33.107259 11244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:33.233883 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 11015
W20260807 02:56:33.265688 11453 connection.cc:535] server connection from 127.0.0.1:45934 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 02:56:33.266192 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 11158
I20260807 02:56:33.267343 11512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:33.389673 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 11158
2026-08-07T02:56:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11150 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:56:33 dist-test-slave-3h5h krb5kdc[12313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:56:33 dist-test-slave-3h5h krb5kdc[12313](info): set up 2 sockets
Aug 07 02:56:33 dist-test-slave-3h5h krb5kdc[12313](info): commencing operation
krb5kdc: starting...
W20260807 02:56:34.720799 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:56:34 dist-test-slave-3h5h krb5kdc[12313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:56:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.126.keytab.
Entry for principal oryx/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.126.keytab.
I20260807 02:56:35.012727 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:43931
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.126.keytab
--principal=oryx/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:43931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:35.384071 12331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:35.384388 12331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:35.384565 12331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:35.392139 12331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:35.392249 12331 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 02:56:35.392354 12331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:35.392465 12331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:35.392556 12331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:35.392673 12331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:35.403153 12331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:43931
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.126.keytab
--principal=oryx/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:43931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:35.404667 12331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:35.406121 12331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:35.419145 12337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:35.419134 12338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:35.419127 12344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:35.419950 12331 server_base.cc:1034] running on GCE node
I20260807 02:56:35.420800 12331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:35.422421 12331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:35.423655 12331 hybrid_clock.cc:648] HybridClock initialized: now 1786071395423592 us; error 34 us; skew 500 ppm
I20260807 02:56:35.423974 12331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:35 dist-test-slave-3h5h krb5kdc[12313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071395, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:35.432928 12331 init.cc:377] Logged in from keytab as oryx/127.1.121.126@KRBTEST.COM (short username oryx)
I20260807 02:56:35.434518 12331 webserver.cc:460] Webserver started at http://127.1.121.126:42455/ using document root <none> and password file <none>
I20260807 02:56:35.435279 12331 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:35.435420 12331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:35.435786 12331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:35.439143 12331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "e8989c159ac64a4dba7df8841fc7b238"
format_stamp: "Formatted at 2026-08-07 02:56:35 on dist-test-slave-3h5h"
server_key: "d26adcfe22da83f05e57205d203bda61"
server_key_iv: "a7b3d82bcd4b1006b616c7ab16f0df46"
server_key_version: "encryptionkey@0"
I20260807 02:56:35.439985 12331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "e8989c159ac64a4dba7df8841fc7b238"
format_stamp: "Formatted at 2026-08-07 02:56:35 on dist-test-slave-3h5h"
server_key: "d26adcfe22da83f05e57205d203bda61"
server_key_iv: "a7b3d82bcd4b1006b616c7ab16f0df46"
server_key_version: "encryptionkey@0"
I20260807 02:56:35.445374 12331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 02:56:35.449003 12354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:35.450459 12331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:56:35.450682 12331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "e8989c159ac64a4dba7df8841fc7b238"
format_stamp: "Formatted at 2026-08-07 02:56:35 on dist-test-slave-3h5h"
server_key: "d26adcfe22da83f05e57205d203bda61"
server_key_iv: "a7b3d82bcd4b1006b616c7ab16f0df46"
server_key_version: "encryptionkey@0"
I20260807 02:56:35.450925 12331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:35.480283 12331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:35.484779 12331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:35.485097 12331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:35.508466 12331 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:43931
I20260807 02:56:35.508478 12436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:43931 every 8 connection(s)
I20260807 02:56:35.509907 12331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:35.513360 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 12331
I20260807 02:56:35.513572 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:35.513989 12437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:35.514369 1509 external_mini_cluster.cc:1428] Setting key f840f6d408f0a9da747d0a770a11f04b
I20260807 02:56:35.529266 12437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238: Bootstrap starting.
Aug 07 02:56:35 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:35.533473 12437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:35.534885 12437 log.cc:824] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:35.538627 12437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238: No bootstrap required, opened a new log
I20260807 02:56:35.540208 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:35.516465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.1.121.126:43931)
0807 02:56:35.517395 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:35.517410 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:35.517473 (+ 63us) server_negotiation.cc:407] Connection header received
0807 02:56:35.519286 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:35.519318 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:35.519856 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:35.520312 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:35.521861 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:35.523794 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:35.525289 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:35.526593 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:35.526797 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:35.530870 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:35.530884 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:35.530911 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:35.530987 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:35.535196 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:35.536208 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:35.536215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:35.536221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:35.536459 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:35.536869 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:35.536875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:35.536878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:35.537668 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:35.537978 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:35.538404 (+ 426us) server_negotiation.cc:299] Negotiation successful
0807 02:56:35.538840 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":403,"threads_started":1}
I20260807 02:56:35.547174 12437 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8989c159ac64a4dba7df8841fc7b238" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43931 } }
I20260807 02:56:35.547497 12437 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:35.547590 12437 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8989c159ac64a4dba7df8841fc7b238, State: Initialized, Role: FOLLOWER
I20260807 02:56:35.548203 12437 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8989c159ac64a4dba7df8841fc7b238" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43931 } }
I20260807 02:56:35.548466 12437 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:35.548635 12437 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:35.548826 12437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:35.551123 12437 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8989c159ac64a4dba7df8841fc7b238" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43931 } }
I20260807 02:56:35.551708 12437 leader_election.cc:302] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8989c159ac64a4dba7df8841fc7b238; no voters:
I20260807 02:56:35.552294 12437 leader_election.cc:288] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:35.552443 12445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:35.553260 12445 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [term 1 LEADER]: Becoming Leader. State: Replica: e8989c159ac64a4dba7df8841fc7b238, State: Running, Role: LEADER
I20260807 02:56:35.553838 12445 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8989c159ac64a4dba7df8841fc7b238" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43931 } }
I20260807 02:56:35.554548 12437 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:35.557399 12447 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8989c159ac64a4dba7df8841fc7b238. Latest consensus state: current_term: 1 leader_uuid: "e8989c159ac64a4dba7df8841fc7b238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8989c159ac64a4dba7df8841fc7b238" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43931 } } }
I20260807 02:56:35.557873 12447 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:35.558491 12446 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8989c159ac64a4dba7df8841fc7b238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8989c159ac64a4dba7df8841fc7b238" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43931 } } }
I20260807 02:56:35.558802 12446 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238 [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:35.562256 12455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:35.566910 12455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:35.575850 12455 catalog_manager.cc:1347] Generated new cluster ID: 409448e24dbb4e0ab028f429bc8a0f3f
I20260807 02:56:35.575978 12455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:35.598534 12455 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:35.599881 12455 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:35.607539 12455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238: Generated new TSK 0
I20260807 02:56:35.608367 12455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab.
Entry for principal oryx/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab.
I20260807 02:56:35.685807 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:43931
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab
--principal=oryx/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:36.074190 12469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:36.074508 12469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:36.074616 12469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:36.082446 12469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:36.082593 12469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:36.082688 12469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:56:36.082782 12469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:56:36.093112 12469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab
--principal=oryx/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.121.126:43931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:36.094498 12469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:36.095913 12469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:36.110296 12477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:36.110266 12480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:36.110819 12482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:36.111692 12469 server_base.cc:1034] running on GCE node
I20260807 02:56:36.112363 12469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:36.113634 12469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:36.114979 12469 hybrid_clock.cc:648] HybridClock initialized: now 1786071396114836 us; error 113 us; skew 500 ppm
I20260807 02:56:36.115370 12469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:36 dist-test-slave-3h5h krb5kdc[12313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071396, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:36.119134 12469 init.cc:377] Logged in from keytab as oryx/127.1.121.65@KRBTEST.COM (short username oryx)
I20260807 02:56:36.120812 12469 webserver.cc:460] Webserver started at http://127.1.121.65:46879/ using document root <none> and password file <none>
I20260807 02:56:36.121691 12469 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:36.121843 12469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:36.122213 12469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:36.125603 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "18d34b988fbe4c0691eb7f8c18e432a1"
format_stamp: "Formatted at 2026-08-07 02:56:36 on dist-test-slave-3h5h"
server_key: "38d745553ab9e9d2d027442c00c86f46"
server_key_iv: "1d4015ded3f189e042707bb61568dd01"
server_key_version: "encryptionkey@0"
I20260807 02:56:36.126646 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "18d34b988fbe4c0691eb7f8c18e432a1"
format_stamp: "Formatted at 2026-08-07 02:56:36 on dist-test-slave-3h5h"
server_key: "38d745553ab9e9d2d027442c00c86f46"
server_key_iv: "1d4015ded3f189e042707bb61568dd01"
server_key_version: "encryptionkey@0"
I20260807 02:56:36.132391 12469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 02:56:36.136547 12496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:36.138188 12469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:36.138396 12469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "18d34b988fbe4c0691eb7f8c18e432a1"
format_stamp: "Formatted at 2026-08-07 02:56:36 on dist-test-slave-3h5h"
server_key: "38d745553ab9e9d2d027442c00c86f46"
server_key_iv: "1d4015ded3f189e042707bb61568dd01"
server_key_version: "encryptionkey@0"
I20260807 02:56:36.138659 12469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:36.160341 12469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:36.165127 12469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:36.165457 12469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:36.168254 12469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:36.168387 12469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:36.168581 12469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:36.168664 12469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:56:36 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071396, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:36.193367 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:36.169108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:51915 (local address 127.1.121.126:43931)
0807 02:56:36.169736 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:36.169744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:36.170968 (+ 1224us) server_negotiation.cc:407] Connection header received
0807 02:56:36.173345 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:36.173351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:36.173451 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:36.173572 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:36.176041 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.177445 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.180154 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.180986 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.181164 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:36.185181 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:36.185190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:36.185197 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:36.185245 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:36.189133 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:36.190153 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:36.190160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:36.190166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:36.190330 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:36.190960 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:36.190969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:36.190975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:36.191459 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:36.191737 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:36.192481 (+ 744us) server_negotiation.cc:299] Negotiation successful
0807 02:56:36.192816 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":343,"threads_started":1}
I20260807 02:56:36.194178 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:36.169836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43931 (local address 127.1.121.65:51915)
0807 02:56:36.170779 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:36.170825 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:36.173031 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:36.173890 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:36.173907 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:36.174731 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:36.175778 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:36.175830 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.177688 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.177693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:36.179935 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:36.179948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.181290 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.181299 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:36.181518 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:36.182240 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:36.182268 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:36.184888 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:36.189283 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:36.189294 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:36.189317 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:36.189944 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:36.190443 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:36.190447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:36.190453 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:36.190805 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:36.191699 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:36.191706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:36.192064 (+ 358us) client_negotiation.cc:769] Sending connection context
0807 02:56:36.192303 (+ 239us) client_negotiation.cc:240] Negotiation successful
0807 02:56:36.192713 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":403,"threads_started":1}
I20260807 02:56:36.222506 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:36.202035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:42749 (local address 127.1.121.126:43931)
0807 02:56:36.202280 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:36.202287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:36.205513 (+ 3226us) server_negotiation.cc:407] Connection header received
0807 02:56:36.205736 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:36.205744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:36.205844 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:36.205953 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:36.207918 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.209222 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.212210 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.212926 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.213038 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:36.215993 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:36.215999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:36.216002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:36.216035 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:36.218715 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:36.219544 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:36.219552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:36.219557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:36.219748 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:36.220353 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:36.220361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:36.220366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:36.220863 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:36.221033 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:36.221711 (+ 678us) server_negotiation.cc:299] Negotiation successful
0807 02:56:36.221991 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 02:56:36.222925 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:36.202074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43931 (local address 127.1.121.65:42749)
0807 02:56:36.205373 (+ 3299us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:36.205394 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:36.205596 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:36.206036 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:36.206045 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:36.206948 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:36.207698 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:36.207722 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.209420 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.209429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:36.211939 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:36.211963 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.213148 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.213161 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:36.213371 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:36.214217 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:36.214251 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:36.215773 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:36.218852 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:36.218860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:36.218865 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:36.219419 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:36.219938 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:36.219949 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:36.219956 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:36.220184 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:36.221043 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:36.221062 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:36.221359 (+ 297us) client_negotiation.cc:769] Sending connection context
0807 02:56:36.221570 (+ 211us) client_negotiation.cc:240] Negotiation successful
0807 02:56:36.222175 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3140,"thread_start_us":2968,"threads_started":1}
I20260807 02:56:36.225319 12469 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:42693
I20260807 02:56:36.225404 12658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:42693 every 8 connection(s)
I20260807 02:56:36.226681 12469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:56:36.228870 12659 heartbeater.cc:347] Connected to a master server at 127.1.121.126:43931
I20260807 02:56:36.229086 12659 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:36.229662 12659 heartbeater.cc:511] Master 127.1.121.126:43931 requested a full tablet report, sending...
I20260807 02:56:36.231200 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 12469
I20260807 02:56:36.231421 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:56:36.232013 12376 ts_manager.cc:194] Registered new tserver with Master: 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693)
I20260807 02:56:36.232334 1509 external_mini_cluster.cc:1428] Setting key 12fd6f7f1093c3f8fa0d6e062ae2456c
I20260807 02:56:36.234000 12376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:51915
WARNING: no policy specified for oryx/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab.
Entry for principal oryx/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab.
I20260807 02:56:36.300276 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:43931
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab
--principal=oryx/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:36.671515 12667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:36.671816 12667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:36.671921 12667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:36.679248 12667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:36.679368 12667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:36.679454 12667 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:56:36.679538 12667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:56:36.689239 12667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab
--principal=oryx/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.121.126:43931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:36.690531 12667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:36.691830 12667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:36.706452 12677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:36.706503 12672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:36.706535 12674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:36.707437 12667 server_base.cc:1034] running on GCE node
I20260807 02:56:36.708092 12667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:36.709129 12667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:36.710399 12667 hybrid_clock.cc:648] HybridClock initialized: now 1786071396710310 us; error 67 us; skew 500 ppm
I20260807 02:56:36.710747 12667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:36 dist-test-slave-3h5h krb5kdc[12313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071396, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:36.714634 12667 init.cc:377] Logged in from keytab as oryx/127.1.121.66@KRBTEST.COM (short username oryx)
I20260807 02:56:36.716440 12667 webserver.cc:460] Webserver started at http://127.1.121.66:44975/ using document root <none> and password file <none>
I20260807 02:56:36.717346 12667 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:36.717502 12667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:36.717896 12667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:36.721211 12667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "3b233536875345b19c467dd322f1f82e"
format_stamp: "Formatted at 2026-08-07 02:56:36 on dist-test-slave-3h5h"
server_key: "aded7333221109a1b5474c7e6b80366f"
server_key_iv: "c6ebf7a2df3d7cc8e56036e3df34150a"
server_key_version: "encryptionkey@0"
I20260807 02:56:36.722100 12667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "3b233536875345b19c467dd322f1f82e"
format_stamp: "Formatted at 2026-08-07 02:56:36 on dist-test-slave-3h5h"
server_key: "aded7333221109a1b5474c7e6b80366f"
server_key_iv: "c6ebf7a2df3d7cc8e56036e3df34150a"
server_key_version: "encryptionkey@0"
I20260807 02:56:36.727912 12667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260807 02:56:36.732084 12689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:36.734064 12667 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260807 02:56:36.734315 12667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "3b233536875345b19c467dd322f1f82e"
format_stamp: "Formatted at 2026-08-07 02:56:36 on dist-test-slave-3h5h"
server_key: "aded7333221109a1b5474c7e6b80366f"
server_key_iv: "c6ebf7a2df3d7cc8e56036e3df34150a"
server_key_version: "encryptionkey@0"
I20260807 02:56:36.734587 12667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:36.749385 12667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:36.753472 12667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:36.753723 12667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:36.756721 12667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:36.756857 12667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:36.757020 12667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:36.757109 12667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:56:36 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071396, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:36.787784 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:36.757541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:57193 (local address 127.1.121.126:43931)
0807 02:56:36.760549 (+ 3008us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:36.760554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:36.760573 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:56:36.763142 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:36.763150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:36.763260 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:36.763419 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:36.766160 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.767449 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.770994 (+ 3545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.771842 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.771994 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:36.778631 (+ 6637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:36.778639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:36.778644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:36.778704 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:36.782887 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:36.784215 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:36.784224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:36.784228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:36.784409 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:36.785129 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:36.785135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:36.785139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:36.785568 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:36.785766 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:36.786991 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0807 02:56:36.787280 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2890,"thread_start_us":2793,"threads_started":1}
I20260807 02:56:36.789814 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:36.758402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43931 (local address 127.1.121.66:57193)
0807 02:56:36.759640 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:36.759698 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:36.762760 (+ 3062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:36.763739 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:36.763756 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:36.764707 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:36.765888 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:36.765933 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.767667 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.767673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:36.770767 (+ 3094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:36.770790 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.772376 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.772387 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:36.772785 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:36.773786 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:36.773823 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:36.778281 (+ 4458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:36.783082 (+ 4801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:36.783099 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:36.783123 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:36.784016 (+ 893us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:36.784560 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:36.784565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:36.784583 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:36.784964 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:36.785758 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:36.785770 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:36.786404 (+ 634us) client_negotiation.cc:769] Sending connection context
0807 02:56:36.786836 (+ 432us) client_negotiation.cc:240] Negotiation successful
0807 02:56:36.787627 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":456,"threads_started":1}
I20260807 02:56:36.803633 12667 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:43617
I20260807 02:56:36.803691 12856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:43617 every 8 connection(s)
I20260807 02:56:36.805101 12667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:56:36.807822 12857 heartbeater.cc:347] Connected to a master server at 127.1.121.126:43931
I20260807 02:56:36.808066 12857 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:36.808737 12857 heartbeater.cc:511] Master 127.1.121.126:43931 requested a full tablet report, sending...
I20260807 02:56:36.810335 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 12667
I20260807 02:56:36.810397 12376 ts_manager.cc:194] Registered new tserver with Master: 3b233536875345b19c467dd322f1f82e (127.1.121.66:43617)
I20260807 02:56:36.810554 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:56:36.811131 1509 external_mini_cluster.cc:1428] Setting key 87c75919083b238b9f6d665441aa1c45
I20260807 02:56:36.811507 12376 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:57193
I20260807 02:56:36.812160 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:36.796084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:47681 (local address 127.1.121.126:43931)
0807 02:56:36.796324 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:36.796334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:36.797378 (+ 1044us) server_negotiation.cc:407] Connection header received
0807 02:56:36.797656 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:36.797662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:36.797758 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:36.797863 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:36.799379 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.800412 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.802305 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.803109 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.803280 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:36.805619 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:36.805630 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:36.805637 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:36.805687 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:36.808888 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:36.809723 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:36.809729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:36.809732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:36.809943 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:36.810435 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:36.810441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:36.810444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:36.810934 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:36.811103 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:36.811600 (+ 497us) server_negotiation.cc:299] Negotiation successful
0807 02:56:36.811855 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260807 02:56:36.812240 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:36.796203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43931 (local address 127.1.121.66:47681)
0807 02:56:36.797255 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:36.797283 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:36.797480 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:36.798011 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:36.798015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:36.798713 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:36.799227 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:36.799245 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.800607 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.800611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:36.802151 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:36.802165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:36.803360 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:36.803372 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:36.803697 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:36.804429 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:36.804448 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:36.805382 (+ 934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:36.809020 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:36.809025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:36.809029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:36.809578 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:36.810107 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:36.810112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:36.810116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:36.810282 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:36.811071 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:36.811076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:36.811268 (+ 192us) client_negotiation.cc:769] Sending connection context
0807 02:56:36.811409 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 02:56:36.811732 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":621,"threads_started":1}
WARNING: no policy specified for oryx/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab.
Entry for principal oryx/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab.
I20260807 02:56:36.876708 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:43931
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab
--principal=oryx/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:56:37.238487 12659 heartbeater.cc:503] Master 127.1.121.126:43931 was elected leader, sending a full tablet report...
W20260807 02:56:37.246799 12866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:37.247064 12866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:37.247174 12866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:37.254467 12866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:37.254575 12866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:37.254695 12866 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:56:37.254839 12866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:56:37.264727 12866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab
--principal=oryx/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.121.126:43931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:37.266081 12866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:37.267416 12866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:37.281659 12872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:37.281627 12873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:37.281589 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:37.283058 12866 server_base.cc:1034] running on GCE node
I20260807 02:56:37.283689 12866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:37.285059 12866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:37.286358 12866 hybrid_clock.cc:648] HybridClock initialized: now 1786071397286254 us; error 63 us; skew 500 ppm
I20260807 02:56:37.286716 12866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:37 dist-test-slave-3h5h krb5kdc[12313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071397, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:37.295912 12866 init.cc:377] Logged in from keytab as oryx/127.1.121.67@KRBTEST.COM (short username oryx)
I20260807 02:56:37.298282 12866 webserver.cc:460] Webserver started at http://127.1.121.67:35379/ using document root <none> and password file <none>
I20260807 02:56:37.299302 12866 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:37.299438 12866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:37.299881 12866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:37.304085 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "aceefefa325e4b1da3c13b5c1b121e8c"
format_stamp: "Formatted at 2026-08-07 02:56:37 on dist-test-slave-3h5h"
server_key: "66623f71f1971486f6623f395f40aad0"
server_key_iv: "82f6c19822495c69a6abd423e2a8a099"
server_key_version: "encryptionkey@0"
I20260807 02:56:37.305047 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "aceefefa325e4b1da3c13b5c1b121e8c"
format_stamp: "Formatted at 2026-08-07 02:56:37 on dist-test-slave-3h5h"
server_key: "66623f71f1971486f6623f395f40aad0"
server_key_iv: "82f6c19822495c69a6abd423e2a8a099"
server_key_version: "encryptionkey@0"
I20260807 02:56:37.311055 12866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 02:56:37.315032 12885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:37.316809 12866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:56:37.317058 12866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "aceefefa325e4b1da3c13b5c1b121e8c"
format_stamp: "Formatted at 2026-08-07 02:56:37 on dist-test-slave-3h5h"
server_key: "66623f71f1971486f6623f395f40aad0"
server_key_iv: "82f6c19822495c69a6abd423e2a8a099"
server_key_version: "encryptionkey@0"
I20260807 02:56:37.317323 12866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:37.352089 12866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:37.357206 12866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:37.357496 12866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:37.360343 12866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:37.360425 12866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:37.360600 12866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:37.360667 12866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:56:37 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071397, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:37.392017 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:37.362293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43931 (local address 127.1.121.67:48953)
0807 02:56:37.363274 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:37.363343 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:37.366685 (+ 3342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:37.367653 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:37.367669 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:37.368584 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:37.369691 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:37.369727 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.371741 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.371748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:37.374269 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:37.374284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.375653 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.375664 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:37.375929 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:37.377375 (+ 1446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:37.377425 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:37.380635 (+ 3210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:37.385799 (+ 5164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:37.385810 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:37.385831 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:37.386550 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:37.387167 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:37.387175 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:37.387182 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:37.387494 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:37.388313 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:37.388326 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:37.388979 (+ 653us) client_negotiation.cc:769] Sending connection context
0807 02:56:37.389341 (+ 362us) client_negotiation.cc:240] Negotiation successful
0807 02:56:37.390064 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":419,"threads_started":1}
I20260807 02:56:37.393822 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:37.361593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:48953 (local address 127.1.121.126:43931)
0807 02:56:37.362098 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:37.362106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:37.363485 (+ 1379us) server_negotiation.cc:407] Connection header received
0807 02:56:37.367113 (+ 3628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:37.367118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:37.367206 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:37.367306 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:37.369954 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.371475 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.374436 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.375333 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.375510 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:37.380942 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:37.380952 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:37.380960 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:37.381016 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:37.385590 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:37.386766 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:37.386773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:37.386780 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:37.386981 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:37.387684 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:37.387691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:37.387697 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:37.388150 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:37.388321 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:37.392883 (+ 4562us) server_negotiation.cc:299] Negotiation successful
0807 02:56:37.393225 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":242,"threads_started":1}
I20260807 02:56:37.424937 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:37.403184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43931 (local address 127.1.121.67:42855)
0807 02:56:37.403955 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:37.403973 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:37.404150 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:37.404550 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:37.404556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:37.405334 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:37.405917 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:37.405931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.407484 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.407503 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:37.409646 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:37.409689 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.410707 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.410714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:37.410903 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:37.414188 (+ 3285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:37.414224 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:37.415532 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:37.420857 (+ 5325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:37.420867 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:37.420877 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:37.421724 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:37.422311 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:37.422318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:37.422324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:37.422596 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:37.423479 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:37.423502 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:37.423834 (+ 332us) client_negotiation.cc:769] Sending connection context
0807 02:56:37.424036 (+ 202us) client_negotiation.cc:240] Negotiation successful
0807 02:56:37.424414 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":402,"threads_started":1}
I20260807 02:56:37.424957 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:37.403062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:42855 (local address 127.1.121.126:43931)
0807 02:56:37.403257 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:37.403261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:37.404074 (+ 813us) server_negotiation.cc:407] Connection header received
0807 02:56:37.404310 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:37.404315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:37.404383 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:37.404499 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:37.406216 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.407318 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.409833 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.410500 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.410665 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:37.415833 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:37.415844 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:37.415852 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:37.415906 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:37.420600 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:37.421931 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:37.421939 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:37.421944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:37.422135 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:37.422784 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:37.422789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:37.422793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:37.423279 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:37.423811 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:37.424202 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 02:56:37.424506 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:56:37.442595 12866 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:43255
I20260807 02:56:37.442641 13054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:43255 every 8 connection(s)
I20260807 02:56:37.444223 12866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:56:37.446856 13056 heartbeater.cc:347] Connected to a master server at 127.1.121.126:43931
I20260807 02:56:37.447145 13056 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:37.447877 13056 heartbeater.cc:511] Master 127.1.121.126:43931 requested a full tablet report, sending...
I20260807 02:56:37.449581 12375 ts_manager.cc:194] Registered new tserver with Master: aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255)
I20260807 02:56:37.450637 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 12866
I20260807 02:56:37.450838 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:56:37.450937 12375 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:48953
I20260807 02:56:37.451489 1509 external_mini_cluster.cc:1428] Setting key 4c48155bdbbd3eacdc481513756a80fa
I20260807 02:56:37.454941 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:56:37 dist-test-slave-3h5h krb5kdc[12313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071397, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:56:37 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:37 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:37.814798 12857 heartbeater.cc:503] Master 127.1.121.126:43931 was elected leader, sending a full tablet report...
I20260807 02:56:38.454231 13056 heartbeater.cc:503] Master 127.1.121.126:43931 was elected leader, sending a full tablet report...
W20260807 02:56:38.653220 13065 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.137s user 0.000s sys 0.022s
W20260807 02:56:38.654635 12896 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:37.509796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.1.121.126:43931)
0807 02:56:37.510017 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:37.510026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:37.510331 (+ 305us) server_negotiation.cc:407] Connection header received
0807 02:56:37.510564 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:37.510576 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:37.510676 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:37.510794 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:37.512240 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.513064 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.514350 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:37.514780 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:37.514929 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:38.654293 (+1139364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33764: BlockingRecv error: recv got EOF from 127.0.0.1:33764 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 07 02:56:38 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071397, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:38.676928 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:38.662522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.1.121.126:43931)
0807 02:56:38.662753 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:38.662758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:38.662927 (+ 169us) server_negotiation.cc:407] Connection header received
0807 02:56:38.663081 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:38.663086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:38.663204 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:38.663296 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:38.664719 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.665503 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.666862 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.667300 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.667400 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:38.671781 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:38.671787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:38.671790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:38.671822 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:38.674464 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:38.675107 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:38.675111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:38.675113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:38.675227 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:38.675561 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:38.675564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:38.675566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:38.675828 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:38.676024 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:38.676388 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 02:56:38.676558 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 02:56:38.681434 12375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:56:38.688364 12375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:56:38.728451 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:38.702012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:42693 (local address 127.0.0.1:50008)
0807 02:56:38.718533 (+ 16521us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:38.718552 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:38.718704 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:38.721558 (+ 2854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:38.721565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:38.721595 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:38.722006 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:38.722019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.723684 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.723690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:38.726084 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:38.726097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.727431 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.727438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:38.727643 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:38.727705 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 02:56:38.727865 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 02:56:38.728073 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16376,"thread_start_us":16321,"threads_started":1}
I20260807 02:56:38.728420 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:38.702533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.1.121.65:42693)
0807 02:56:38.720630 (+ 18097us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:38.720639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:38.720667 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:56:38.720767 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:38.720777 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:38.721081 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:38.721304 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:38.722205 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.723512 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.726240 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.727208 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.727372 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:38.727616 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:38.727913 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 02:56:38.728101 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17901,"thread_start_us":17720,"threads_started":1}
I20260807 02:56:38.728997 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:38.701417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:43255 (local address 127.0.0.1:35858)
0807 02:56:38.716748 (+ 15331us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:38.716808 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:38.717153 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:38.719855 (+ 2702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:38.719860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:38.719893 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:38.720450 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:38.720471 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.723031 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.723037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:38.725641 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:38.725655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.727911 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.727920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:38.728223 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:38.728357 (+ 134us) client_negotiation.cc:769] Sending connection context
0807 02:56:38.728442 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 02:56:38.728597 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14896,"mutex_wait_us":118,"thread_start_us":15999,"threads_started":1}
I20260807 02:56:38.729423 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:38.702408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.1.121.66:43617)
0807 02:56:38.711429 (+ 9021us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:38.711444 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:38.718203 (+ 6759us) server_negotiation.cc:407] Connection header received
0807 02:56:38.718707 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:38.718720 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:38.719049 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:38.719279 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:38.720755 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.722387 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.725757 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.727370 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.727489 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:38.727693 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:38.728548 (+ 855us) server_negotiation.cc:299] Negotiation successful
0807 02:56:38.729106 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8776,"thread_start_us":8579,"threads_started":1}
I20260807 02:56:38.729490 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:38.701641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.1.121.67:43255)
0807 02:56:38.719338 (+ 17697us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:38.719347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:38.719383 (+ 36us) server_negotiation.cc:407] Connection header received
0807 02:56:38.719466 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:38.719471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:38.719645 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:38.719859 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:38.721011 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.722873 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.725852 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.727339 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.727487 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:38.727705 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:38.728987 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0807 02:56:38.729188 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17534,"thread_start_us":17413,"threads_started":1}
I20260807 02:56:38.733094 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:38.701734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:43617 (local address 127.0.0.1:35514)
0807 02:56:38.718058 (+ 16324us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:38.718085 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:38.718222 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:38.719651 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:38.719662 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:38.719703 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:38.720478 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:38.720494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.722532 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.722539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:38.725595 (+ 3056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:38.725616 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:38.727929 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:38.727935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:38.728227 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:38.728361 (+ 134us) client_negotiation.cc:769] Sending connection context
0807 02:56:38.729907 (+ 1546us) client_negotiation.cc:240] Negotiation successful
0807 02:56:38.730037 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16204,"mutex_wait_us":251,"thread_start_us":16059,"threads_started":1}
I20260807 02:56:38.733386 12569 tablet_service.cc:1467] Processing CreateTablet for tablet a51fc29687f7400bb4b26ee40d364ec1 (DEFAULT_TABLE table=test-table [id=74c828c4c1ca40d8a576f31bf731c925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:38.733385 12952 tablet_service.cc:1467] Processing CreateTablet for tablet a51fc29687f7400bb4b26ee40d364ec1 (DEFAULT_TABLE table=test-table [id=74c828c4c1ca40d8a576f31bf731c925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:38.735028 12569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a51fc29687f7400bb4b26ee40d364ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:38.735029 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a51fc29687f7400bb4b26ee40d364ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:38.736374 12762 tablet_service.cc:1467] Processing CreateTablet for tablet a51fc29687f7400bb4b26ee40d364ec1 (DEFAULT_TABLE table=test-table [id=74c828c4c1ca40d8a576f31bf731c925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:38.737702 12762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a51fc29687f7400bb4b26ee40d364ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:38.750499 13085 tablet_bootstrap.cc:492] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: Bootstrap starting.
I20260807 02:56:38.754029 13087 tablet_bootstrap.cc:492] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: Bootstrap starting.
I20260807 02:56:38.757982 13085 tablet_bootstrap.cc:654] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:38.758380 13086 tablet_bootstrap.cc:492] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: Bootstrap starting.
I20260807 02:56:38.759296 13087 tablet_bootstrap.cc:654] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:38.759450 13085 log.cc:824] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:38.760478 13087 log.cc:824] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:38.761809 13086 tablet_bootstrap.cc:654] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:38.762950 13086 log.cc:824] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:38.763914 13085 tablet_bootstrap.cc:492] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: No bootstrap required, opened a new log
I20260807 02:56:38.764266 13085 ts_tablet_manager.cc:1397] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20260807 02:56:38.765709 13086 tablet_bootstrap.cc:492] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: No bootstrap required, opened a new log
I20260807 02:56:38.765976 13086 ts_tablet_manager.cc:1397] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 02:56:38.767355 13087 tablet_bootstrap.cc:492] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: No bootstrap required, opened a new log
I20260807 02:56:38.767627 13087 ts_tablet_manager.cc:1397] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260807 02:56:38.773260 13086 raft_consensus.cc:348] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } }
I20260807 02:56:38.773633 13086 raft_consensus.cc:374] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:38.773768 13086 raft_consensus.cc:729] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b233536875345b19c467dd322f1f82e, State: Initialized, Role: FOLLOWER
I20260807 02:56:38.774408 13086 consensus_queue.cc:260] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } }
I20260807 02:56:38.775434 13085 raft_consensus.cc:348] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } }
I20260807 02:56:38.775880 13085 raft_consensus.cc:374] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:38.775986 13085 raft_consensus.cc:729] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d34b988fbe4c0691eb7f8c18e432a1, State: Initialized, Role: FOLLOWER
I20260807 02:56:38.776166 13086 ts_tablet_manager.cc:1428] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260807 02:56:38.776712 13085 consensus_queue.cc:260] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } }
I20260807 02:56:38.780398 13085 ts_tablet_manager.cc:1428] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260807 02:56:38.780915 13087 raft_consensus.cc:348] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } }
I20260807 02:56:38.781381 13087 raft_consensus.cc:374] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:38.781536 13087 raft_consensus.cc:729] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aceefefa325e4b1da3c13b5c1b121e8c, State: Initialized, Role: FOLLOWER
I20260807 02:56:38.782208 13087 consensus_queue.cc:260] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } }
I20260807 02:56:38.784049 13087 ts_tablet_manager.cc:1428] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260807 02:56:38.808379 12858 tablet.cc:2369] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:56:38.947492 13057 tablet.cc:2369] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:56:38.980971 12663 tablet.cc:2369] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:56:39.017581 13094 raft_consensus.cc:482] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:39.018002 13094 raft_consensus.cc:504] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } }
I20260807 02:56:39.019853 13094 leader_election.cc:288] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255), 3b233536875345b19c467dd322f1f82e (127.1.121.66:43617)
I20260807 02:56:39.027453 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.020428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:46415 (local address 127.1.121.66:43617)
0807 02:56:39.020713 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.020723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.021742 (+ 1019us) server_negotiation.cc:407] Connection header received
0807 02:56:39.022101 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.022112 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.022185 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.022286 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:39.022882 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.023635 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.025719 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.026502 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.026707 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.026787 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.027060 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.027193 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:56:39.027607 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.020285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:43617 (local address 127.1.121.65:46415)
0807 02:56:39.021616 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:39.021632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:39.021930 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:39.022304 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:39.022308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:39.022342 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:39.022705 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.022716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.023768 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.023777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.025550 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.025567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.026723 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.026729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.026892 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.026916 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:56:39.027022 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 02:56:39.027177 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"mutex_wait_us":49,"thread_start_us":607,"threads_started":1}
I20260807 02:56:39.027910 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.020357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:35005 (local address 127.1.121.67:43255)
0807 02:56:39.020617 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.020628 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.021741 (+ 1113us) server_negotiation.cc:407] Connection header received
0807 02:56:39.022206 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.022213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.022348 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.022452 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:39.023126 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.023930 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.025987 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.026864 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.027079 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.027163 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.027481 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.027630 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 02:56:39.028119 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.020184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:43255 (local address 127.1.121.65:35005)
0807 02:56:39.021608 (+ 1424us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:39.021642 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:39.021984 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:39.022478 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:39.022483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:39.022516 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:39.022961 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.022971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.024108 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.024114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.025813 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.025828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.027126 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.027134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.027297 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.027326 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 02:56:39.027438 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 02:56:39.027586 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":4,"spinlock_wait_cycles":159104,"thread_start_us":990,"threads_started":1}
I20260807 02:56:39.029026 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a51fc29687f7400bb4b26ee40d364ec1" candidate_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b233536875345b19c467dd322f1f82e" is_pre_election: true
I20260807 02:56:39.029290 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a51fc29687f7400bb4b26ee40d364ec1" candidate_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" is_pre_election: true
I20260807 02:56:39.029646 12789 raft_consensus.cc:2454] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d34b988fbe4c0691eb7f8c18e432a1 in term 0.
I20260807 02:56:39.029838 12990 raft_consensus.cc:2454] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18d34b988fbe4c0691eb7f8c18e432a1 in term 0.
I20260807 02:56:39.030471 12501 leader_election.cc:302] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18d34b988fbe4c0691eb7f8c18e432a1, 3b233536875345b19c467dd322f1f82e, aceefefa325e4b1da3c13b5c1b121e8c; no voters:
I20260807 02:56:39.031220 13094 raft_consensus.cc:2781] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:56:39.031421 13094 raft_consensus.cc:482] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:56:39.031574 13094 raft_consensus.cc:3037] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:39.034086 13094 raft_consensus.cc:504] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } }
I20260807 02:56:39.034787 13094 leader_election.cc:288] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [CANDIDATE]: Term 1 election: Requested vote from peers aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255), 3b233536875345b19c467dd322f1f82e (127.1.121.66:43617)
I20260807 02:56:39.035199 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a51fc29687f7400bb4b26ee40d364ec1" candidate_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aceefefa325e4b1da3c13b5c1b121e8c"
I20260807 02:56:39.035416 12990 raft_consensus.cc:3037] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:39.035346 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a51fc29687f7400bb4b26ee40d364ec1" candidate_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b233536875345b19c467dd322f1f82e"
I20260807 02:56:39.035619 12789 raft_consensus.cc:3037] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:39.037849 12990 raft_consensus.cc:2454] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d34b988fbe4c0691eb7f8c18e432a1 in term 1.
I20260807 02:56:39.038295 12789 raft_consensus.cc:2454] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d34b988fbe4c0691eb7f8c18e432a1 in term 1.
I20260807 02:56:39.038475 12497 leader_election.cc:302] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d34b988fbe4c0691eb7f8c18e432a1, aceefefa325e4b1da3c13b5c1b121e8c; no voters:
I20260807 02:56:39.038856 13094 raft_consensus.cc:2781] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:39.039371 13094 raft_consensus.cc:686] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 LEADER]: Becoming Leader. State: Replica: 18d34b988fbe4c0691eb7f8c18e432a1, State: Running, Role: LEADER
I20260807 02:56:39.039919 13094 consensus_queue.cc:237] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } }
I20260807 02:56:39.045607 12374 catalog_manager.cc:5495] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65). New cstate: current_term: 1 leader_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } health_report { overall_health: HEALTHY } } }
I20260807 02:56:39.103152 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.097486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.1.121.65:42693)
0807 02:56:39.097747 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.097753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.097868 (+ 115us) server_negotiation.cc:407] Connection header received
0807 02:56:39.098172 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.098176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.098272 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.098434 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:56:39.098821 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.099554 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.100957 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.101371 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.101509 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.101979 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:56:39.102526 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:56:39.102693 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.102786 (+ 93us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.102872 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 02:56:39.111915 12789 raft_consensus.cc:1261] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Refusing update from remote peer 18d34b988fbe4c0691eb7f8c18e432a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:39.111914 12990 raft_consensus.cc:1261] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Refusing update from remote peer 18d34b988fbe4c0691eb7f8c18e432a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:39.113121 13098 consensus_queue.cc:1035] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:39.113525 13094 consensus_queue.cc:1035] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:39.123275 13101 mvcc.cc:205] Tried to move back new op lower bound from 7315748450751426560 to 7315748450472034304. Current Snapshot: MvccSnapshot[applied={T|T < 7315748450751426560}]
I20260807 02:56:39.140329 12374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:39.140616 12374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:39.144220 12374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238: Sending DeleteTablet for 3 replicas of tablet a51fc29687f7400bb4b26ee40d364ec1
I20260807 02:56:39.145646 12569 tablet_service.cc:1514] Processing DeleteTablet for tablet a51fc29687f7400bb4b26ee40d364ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:39 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:50008
I20260807 02:56:39.145684 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet a51fc29687f7400bb4b26ee40d364ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:39 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:35858
I20260807 02:56:39.146522 12762 tablet_service.cc:1514] Processing DeleteTablet for tablet a51fc29687f7400bb4b26ee40d364ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:39 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:35514
I20260807 02:56:39.146934 13113 tablet_replica.cc:330] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: stopping tablet replica
I20260807 02:56:39.147436 13113 raft_consensus.cc:2229] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:39.147781 13115 tablet_replica.cc:330] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: stopping tablet replica
I20260807 02:56:39.148136 13113 raft_consensus.cc:2258] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:39.148485 13115 raft_consensus.cc:2229] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:39.149206 13115 raft_consensus.cc:2258] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:39.151217 13115 ts_tablet_manager.cc:1905] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:39.151217 13113 ts_tablet_manager.cc:1905] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:39.153302 12374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:56:39.154115 12374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:56:39.159111 13115 ts_tablet_manager.cc:1918] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:39.159404 13115 log.cc:1196] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/a51fc29687f7400bb4b26ee40d364ec1
I20260807 02:56:39.160097 13115 ts_tablet_manager.cc:1939] T a51fc29687f7400bb4b26ee40d364ec1 P 3b233536875345b19c467dd322f1f82e: Deleting consensus metadata
I20260807 02:56:39.163136 12361 catalog_manager.cc:4841] TS 3b233536875345b19c467dd322f1f82e (127.1.121.66:43617): tablet a51fc29687f7400bb4b26ee40d364ec1 (table test-table [id=74c828c4c1ca40d8a576f31bf731c925]) successfully deleted
I20260807 02:56:39.164933 12762 tablet_service.cc:1467] Processing CreateTablet for tablet d87bc91e728f47dfb2f79c5f2021daa4 (DEFAULT_TABLE table=test-table [id=4364f0732ac24394b1ba88cfb173487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:39.164934 12952 tablet_service.cc:1467] Processing CreateTablet for tablet d87bc91e728f47dfb2f79c5f2021daa4 (DEFAULT_TABLE table=test-table [id=4364f0732ac24394b1ba88cfb173487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:39.164984 12567 tablet_service.cc:1467] Processing CreateTablet for tablet d87bc91e728f47dfb2f79c5f2021daa4 (DEFAULT_TABLE table=test-table [id=4364f0732ac24394b1ba88cfb173487b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:39.165390 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87bc91e728f47dfb2f79c5f2021daa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:39.165390 12762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87bc91e728f47dfb2f79c5f2021daa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:39.165843 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87bc91e728f47dfb2f79c5f2021daa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:39.166664 13112 tablet_replica.cc:330] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: stopping tablet replica
I20260807 02:56:39.167171 13112 raft_consensus.cc:2229] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:56:39.167912 13112 raft_consensus.cc:2258] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:39.170228 13112 ts_tablet_manager.cc:1905] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:39.171914 13087 tablet_bootstrap.cc:492] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: Bootstrap starting.
I20260807 02:56:39.173723 13085 tablet_bootstrap.cc:492] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: Bootstrap starting.
I20260807 02:56:39.173843 13087 tablet_bootstrap.cc:654] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:39.175429 13085 tablet_bootstrap.cc:654] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:39.176569 13086 tablet_bootstrap.cc:492] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: Bootstrap starting.
I20260807 02:56:39.177141 13113 ts_tablet_manager.cc:1918] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:39.177721 13113 log.cc:1196] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/a51fc29687f7400bb4b26ee40d364ec1
I20260807 02:56:39.178368 13113 ts_tablet_manager.cc:1939] T a51fc29687f7400bb4b26ee40d364ec1 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting consensus metadata
I20260807 02:56:39.178378 13086 tablet_bootstrap.cc:654] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:39.181849 12355 catalog_manager.cc:4841] TS aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255): tablet a51fc29687f7400bb4b26ee40d364ec1 (table test-table [id=74c828c4c1ca40d8a576f31bf731c925]) successfully deleted
I20260807 02:56:39.184695 13085 tablet_bootstrap.cc:492] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: No bootstrap required, opened a new log
I20260807 02:56:39.184993 13085 ts_tablet_manager.cc:1397] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20260807 02:56:39.185402 13087 tablet_bootstrap.cc:492] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: No bootstrap required, opened a new log
I20260807 02:56:39.185583 13087 ts_tablet_manager.cc:1397] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260807 02:56:39.185949 13086 tablet_bootstrap.cc:492] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: No bootstrap required, opened a new log
I20260807 02:56:39.185976 13085 raft_consensus.cc:348] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.186156 13086 ts_tablet_manager.cc:1397] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 02:56:39.186230 13085 raft_consensus.cc:374] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:39.186336 13085 raft_consensus.cc:729] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d34b988fbe4c0691eb7f8c18e432a1, State: Initialized, Role: FOLLOWER
I20260807 02:56:39.186514 13087 raft_consensus.cc:348] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.186751 13087 raft_consensus.cc:374] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:39.186669 13085 consensus_queue.cc:260] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.186846 13087 raft_consensus.cc:729] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aceefefa325e4b1da3c13b5c1b121e8c, State: Initialized, Role: FOLLOWER
I20260807 02:56:39.186838 13086 raft_consensus.cc:348] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.187114 13086 raft_consensus.cc:374] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:39.187215 13086 raft_consensus.cc:729] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b233536875345b19c467dd322f1f82e, State: Initialized, Role: FOLLOWER
I20260807 02:56:39.187110 13087 consensus_queue.cc:260] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.187428 13085 ts_tablet_manager.cc:1428] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 02:56:39.187533 13086 consensus_queue.cc:260] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.187783 13087 ts_tablet_manager.cc:1428] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 02:56:39.188297 13086 ts_tablet_manager.cc:1428] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 02:56:39.189672 13112 ts_tablet_manager.cc:1918] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:39.189879 13112 log.cc:1196] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/a51fc29687f7400bb4b26ee40d364ec1
I20260807 02:56:39.190562 13112 ts_tablet_manager.cc:1939] T a51fc29687f7400bb4b26ee40d364ec1 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting consensus metadata
I20260807 02:56:39.192718 12360 catalog_manager.cc:4841] TS 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693): tablet a51fc29687f7400bb4b26ee40d364ec1 (table test-table [id=74c828c4c1ca40d8a576f31bf731c925]) successfully deleted
I20260807 02:56:39.239991 13091 raft_consensus.cc:482] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:39.240257 13091 raft_consensus.cc:504] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.242352 13091 leader_election.cc:288] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693), aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255)
I20260807 02:56:39.263908 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.242930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:33789 (local address 127.1.121.65:42693)
0807 02:56:39.243148 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.243157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.253266 (+ 10109us) server_negotiation.cc:407] Connection header received
0807 02:56:39.254002 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.254012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.254115 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.254206 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:39.255682 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.257126 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.260142 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.261627 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.261779 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.261902 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.263328 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.263529 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:56:39.264555 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.244115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:46863 (local address 127.1.121.67:43255)
0807 02:56:39.244351 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.244358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.256168 (+ 11810us) server_negotiation.cc:407] Connection header received
0807 02:56:39.256559 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.256566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.256664 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.256732 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:39.257743 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.259006 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.261752 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.263148 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.263306 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.263380 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.264089 (+ 709us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.264261 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 02:56:39.265064 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.242726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:42693 (local address 127.1.121.66:33789)
0807 02:56:39.253093 (+ 10367us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:39.253127 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:39.253748 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:39.254609 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:39.254618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:39.254658 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:39.255198 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.255218 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.257340 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.257351 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.259931 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.259947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.262890 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.262897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.263097 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.263136 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 02:56:39.264324 (+ 1188us) client_negotiation.cc:240] Negotiation successful
0807 02:56:39.264489 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10148,"mutex_wait_us":33,"thread_start_us":10569,"threads_started":1}
I20260807 02:56:39.265523 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.243434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:43255 (local address 127.1.121.66:46863)
0807 02:56:39.256046 (+ 12612us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:39.256068 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:39.256349 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:39.256992 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:39.256998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:39.257024 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:39.257513 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.257533 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.259204 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.259210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.261542 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.261561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.263605 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.263622 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.263822 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.263873 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 02:56:39.264799 (+ 926us) client_negotiation.cc:240] Negotiation successful
0807 02:56:39.264991 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12454,"mutex_wait_us":5,"thread_start_us":11803,"threads_started":1}
I20260807 02:56:39.266198 12600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87bc91e728f47dfb2f79c5f2021daa4" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" is_pre_election: true
I20260807 02:56:39.266373 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87bc91e728f47dfb2f79c5f2021daa4" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" is_pre_election: true
I20260807 02:56:39.266688 12990 raft_consensus.cc:2454] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 0.
I20260807 02:56:39.266700 12600 raft_consensus.cc:2454] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 0.
I20260807 02:56:39.267303 12692 leader_election.cc:302] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b233536875345b19c467dd322f1f82e, aceefefa325e4b1da3c13b5c1b121e8c; no voters:
I20260807 02:56:39.267853 13091 raft_consensus.cc:2781] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:56:39.268013 13091 raft_consensus.cc:482] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:56:39.268134 13091 raft_consensus.cc:3037] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:39.270622 13091 raft_consensus.cc:504] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.271309 13091 leader_election.cc:288] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 election: Requested vote from peers 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693), aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255)
I20260807 02:56:39.271806 12600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87bc91e728f47dfb2f79c5f2021daa4" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d34b988fbe4c0691eb7f8c18e432a1"
I20260807 02:56:39.272047 12600 raft_consensus.cc:3037] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:39.271978 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87bc91e728f47dfb2f79c5f2021daa4" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aceefefa325e4b1da3c13b5c1b121e8c"
I20260807 02:56:39.272207 12990 raft_consensus.cc:3037] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:39.275310 12990 raft_consensus.cc:2454] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 1.
I20260807 02:56:39.275480 12600 raft_consensus.cc:2454] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 1.
I20260807 02:56:39.275872 12692 leader_election.cc:302] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b233536875345b19c467dd322f1f82e, aceefefa325e4b1da3c13b5c1b121e8c; no voters:
I20260807 02:56:39.276247 13091 raft_consensus.cc:2781] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:39.276800 13091 raft_consensus.cc:686] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 1 LEADER]: Becoming Leader. State: Replica: 3b233536875345b19c467dd322f1f82e, State: Running, Role: LEADER
I20260807 02:56:39.277451 13091 consensus_queue.cc:237] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.281265 12375 catalog_manager.cc:5495] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b233536875345b19c467dd322f1f82e (127.1.121.66). New cstate: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } health_report { overall_health: UNKNOWN } } }
Aug 07 02:56:39 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071395, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.126@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:39.353731 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.335914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43931 (local address 127.0.0.1:34806)
0807 02:56:39.336674 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:39.336698 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:39.336946 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:39.337405 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:39.337414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:39.338380 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:39.338686 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.338700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.339948 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.339954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.341733 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.341779 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.342593 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.342605 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.342814 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.343594 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:39.343628 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:39.346606 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:39.351034 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:39.351046 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:39.351052 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:39.351467 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:39.351977 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:39.351980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:39.351982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:39.352135 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:39.352884 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:39.352895 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:39.353055 (+ 160us) client_negotiation.cc:769] Sending connection context
0807 02:56:39.353200 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 02:56:39.353410 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":358,"threads_started":1}
I20260807 02:56:39.354125 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.335945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.1.121.126:43931)
0807 02:56:39.337086 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.337090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.337110 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:56:39.337155 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.337159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.337222 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.337311 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:39.338836 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.339808 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.341957 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.342394 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.342501 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.346813 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:39.346821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:39.346827 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:39.346868 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:39.350891 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:39.351614 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:39.351623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:39.351628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:39.351856 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:39.352263 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:39.352269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:39.352273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:39.352692 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:39.352901 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.353371 (+ 470us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.353642 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":890,"threads_started":1}
I20260807 02:56:39.360428 12375 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:34806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 02:56:39.361868 12375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:56:39.369164 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 03778af824014b83b3aab83de3832796 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46d78491f14748a6b8b075f65070b0c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:56:39.369357 12762 tablet_service.cc:1467] Processing CreateTablet for tablet 03778af824014b83b3aab83de3832796 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46d78491f14748a6b8b075f65070b0c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:56:39.369837 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03778af824014b83b3aab83de3832796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:39.369959 12762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03778af824014b83b3aab83de3832796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:39.373092 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 03778af824014b83b3aab83de3832796 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46d78491f14748a6b8b075f65070b0c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:56:39.373706 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03778af824014b83b3aab83de3832796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:39.376642 13086 tablet_bootstrap.cc:492] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: Bootstrap starting.
I20260807 02:56:39.378130 13085 tablet_bootstrap.cc:492] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: Bootstrap starting.
I20260807 02:56:39.378505 13086 tablet_bootstrap.cc:654] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:39.380077 13085 tablet_bootstrap.cc:654] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:39.382627 13087 tablet_bootstrap.cc:492] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: Bootstrap starting.
I20260807 02:56:39.384925 13087 tablet_bootstrap.cc:654] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:39.389639 13086 tablet_bootstrap.cc:492] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: No bootstrap required, opened a new log
I20260807 02:56:39.389707 13087 tablet_bootstrap.cc:492] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: No bootstrap required, opened a new log
I20260807 02:56:39.389902 13087 ts_tablet_manager.cc:1397] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 02:56:39.389952 13086 ts_tablet_manager.cc:1397] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20260807 02:56:39.390745 13087 raft_consensus.cc:348] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.391026 13087 raft_consensus.cc:374] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:39.391120 13087 raft_consensus.cc:729] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aceefefa325e4b1da3c13b5c1b121e8c, State: Initialized, Role: FOLLOWER
I20260807 02:56:39.390908 13086 raft_consensus.cc:348] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.391448 13086 raft_consensus.cc:374] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:39.391489 13087 consensus_queue.cc:260] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.391628 13086 raft_consensus.cc:729] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b233536875345b19c467dd322f1f82e, State: Initialized, Role: FOLLOWER
I20260807 02:56:39.391949 13086 consensus_queue.cc:260] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.392299 13087 ts_tablet_manager.cc:1428] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260807 02:56:39.392127 13095 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.392633 13095 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:56:39.392886 13086 ts_tablet_manager.cc:1428] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 02:56:39.392760 13091 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.393200 13091 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:56:39.394405 13085 tablet_bootstrap.cc:492] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: No bootstrap required, opened a new log
I20260807 02:56:39.394624 13085 ts_tablet_manager.cc:1397] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20260807 02:56:39.395351 13085 raft_consensus.cc:348] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.395797 13085 raft_consensus.cc:374] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:39.396807 13085 raft_consensus.cc:729] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d34b988fbe4c0691eb7f8c18e432a1, State: Initialized, Role: FOLLOWER
I20260807 02:56:39.397205 13085 consensus_queue.cc:260] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.397765 13098 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.398084 13098 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:56:39.399248 13085 ts_tablet_manager.cc:1428] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260807 02:56:39.446743 13091 raft_consensus.cc:482] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:39.447026 13091 raft_consensus.cc:504] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.447867 13091 leader_election.cc:288] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693), aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255)
I20260807 02:56:39.448772 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03778af824014b83b3aab83de3832796" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" is_pre_election: true
I20260807 02:56:39.448812 12600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03778af824014b83b3aab83de3832796" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" is_pre_election: true
I20260807 02:56:39.449133 12990 raft_consensus.cc:2454] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 0.
I20260807 02:56:39.449186 12600 raft_consensus.cc:2454] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 0.
I20260807 02:56:39.449731 12692 leader_election.cc:302] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b233536875345b19c467dd322f1f82e, aceefefa325e4b1da3c13b5c1b121e8c; no voters:
I20260807 02:56:39.450233 13091 raft_consensus.cc:2781] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:56:39.450350 13091 raft_consensus.cc:482] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:56:39.450496 13091 raft_consensus.cc:3037] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:39.454108 13091 raft_consensus.cc:504] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.455628 12600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03778af824014b83b3aab83de3832796" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d34b988fbe4c0691eb7f8c18e432a1"
I20260807 02:56:39.455919 12600 raft_consensus.cc:3037] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:39.455935 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03778af824014b83b3aab83de3832796" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aceefefa325e4b1da3c13b5c1b121e8c"
I20260807 02:56:39.456224 12990 raft_consensus.cc:3037] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:39.459786 12600 raft_consensus.cc:2454] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 1.
I20260807 02:56:39.459810 12990 raft_consensus.cc:2454] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 1.
I20260807 02:56:39.460335 12692 leader_election.cc:302] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b233536875345b19c467dd322f1f82e, aceefefa325e4b1da3c13b5c1b121e8c; no voters:
I20260807 02:56:39.460722 13123 raft_consensus.cc:2781] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:39.464993 13091 leader_election.cc:288] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 election: Requested vote from peers 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693), aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255)
I20260807 02:56:39.466101 13123 raft_consensus.cc:686] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [term 1 LEADER]: Becoming Leader. State: Replica: 3b233536875345b19c467dd322f1f82e, State: Running, Role: LEADER
I20260807 02:56:39.466531 13123 consensus_queue.cc:237] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.470237 13091 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: TxnStatusTablet state changed. Reason: New leader 3b233536875345b19c467dd322f1f82e. Latest consensus state: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.470597 13091 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:56:39.471293 13142 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 02:56:39.471628 13142 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 02:56:39.472265 12376 catalog_manager.cc:5495] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b233536875345b19c467dd322f1f82e (127.1.121.66). New cstate: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } health_report { overall_health: UNKNOWN } } }
I20260807 02:56:39.502995 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.494082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:43617 (local address 127.0.0.1:35422)
0807 02:56:39.494349 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:39.494372 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:39.494617 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:39.495613 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:39.495621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:39.495639 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:56:39.495854 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.495866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.497707 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.497714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.499521 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.499539 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.500671 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.500679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.501003 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.501093 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:56:39.502284 (+ 1191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:56:39.502311 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 02:56:39.502412 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 02:56:39.502599 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 02:56:39.503013 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.494063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.1.121.66:43617)
0807 02:56:39.494281 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.494288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.494504 (+ 216us) server_negotiation.cc:407] Connection header received
0807 02:56:39.495188 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.495195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.495326 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.495525 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:56:39.496030 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.497545 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.499706 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.500462 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.500632 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.501352 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:56:39.502127 (+ 775us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:56:39.502311 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.502447 (+ 136us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.502550 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 02:56:39.508518 12990 raft_consensus.cc:1261] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Refusing update from remote peer 3b233536875345b19c467dd322f1f82e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:39.509665 13091 consensus_queue.cc:1035] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:39.512683 12600 raft_consensus.cc:1261] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Refusing update from remote peer 3b233536875345b19c467dd322f1f82e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:39.514087 13091 consensus_queue.cc:1035] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:39.529368 13098 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: TxnStatusTablet state changed. Reason: New leader 3b233536875345b19c467dd322f1f82e. Latest consensus state: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.529765 13098 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:56:39.531270 13095 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: TxnStatusTablet state changed. Reason: New leader 3b233536875345b19c467dd322f1f82e. Latest consensus state: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.531651 13095 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:56:39.533221 13091 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.533587 13091 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:56:39.534041 13102 mvcc.cc:205] Tried to move back new op lower bound from 7315748452376944640 to 7315748452218413056. Current Snapshot: MvccSnapshot[applied={T|T < 7315748452376944640 or (T in {7315748452376944640})}]
I20260807 02:56:39.536134 13140 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.536629 13140 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P 3b233536875345b19c467dd322f1f82e: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:56:39.542105 13098 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.542459 13098 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P 18d34b988fbe4c0691eb7f8c18e432a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:56:39.544080 13095 tablet_replica.cc:439] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } } }
I20260807 02:56:39.544327 13095 tablet_replica.cc:442] T 03778af824014b83b3aab83de3832796 P aceefefa325e4b1da3c13b5c1b121e8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:56:39.550017 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.541654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.1.121.66:43617)
0807 02:56:39.541843 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.541851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.541870 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:56:39.543160 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.543167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.543264 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.543369 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:56:39.544068 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.545226 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.547034 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.547700 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.547830 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.548785 (+ 955us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:56:39.549009 (+ 224us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:56:39.549100 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.549601 (+ 501us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.549692 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 02:56:39.569077 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.562439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:60781 (local address 127.1.121.66:43617)
0807 02:56:39.562687 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.562695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.563450 (+ 755us) server_negotiation.cc:407] Connection header received
0807 02:56:39.563694 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.563700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.563771 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.563875 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:56:39.564503 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.565392 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.567011 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.567519 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.567672 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.568132 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:56:39.568261 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:56:39.568367 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.568553 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.568634 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 02:56:39.569216 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.562464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:43617 (local address 127.1.121.66:60781)
0807 02:56:39.563343 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:39.563360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:39.563546 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:39.563902 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:39.563906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:39.563918 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:56:39.564325 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.564334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.565541 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.565548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.566836 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:39.566853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.567725 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.567734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:39.567883 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.567936 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:56:39.568368 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:56:39.568390 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 02:56:39.568530 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 02:56:39.568693 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":4,"thread_start_us":540,"threads_started":1}
I20260807 02:56:39.580089 12988 raft_consensus.cc:1261] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Refusing update from remote peer 3b233536875345b19c467dd322f1f82e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:39.580163 12600 raft_consensus.cc:1261] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Refusing update from remote peer 3b233536875345b19c467dd322f1f82e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:39.580821 13091 consensus_queue.cc:1035] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:39.581194 13123 consensus_queue.cc:1035] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:39.599334 13104 mvcc.cc:205] Tried to move back new op lower bound from 7315748452674916352 to 7315748451444768768. Current Snapshot: MvccSnapshot[applied={T|T < 7315748452674916352}]
I20260807 02:56:39.672104 12376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:39.672425 12376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:39.676285 12376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238: Sending DeleteTablet for 3 replicas of tablet d87bc91e728f47dfb2f79c5f2021daa4
I20260807 02:56:39.677827 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet d87bc91e728f47dfb2f79c5f2021daa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:39 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:35858
I20260807 02:56:39.677913 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet d87bc91e728f47dfb2f79c5f2021daa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:39 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:35514
I20260807 02:56:39.678215 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet d87bc91e728f47dfb2f79c5f2021daa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:39 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:50008
I20260807 02:56:39.678576 13113 tablet_replica.cc:330] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: stopping tablet replica
I20260807 02:56:39.678798 13113 raft_consensus.cc:2229] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:39.679008 13113 raft_consensus.cc:2258] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:39.679302 13112 tablet_replica.cc:330] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: stopping tablet replica
I20260807 02:56:39.679533 13112 raft_consensus.cc:2229] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:39.679738 13112 raft_consensus.cc:2258] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:39.682289 13112 ts_tablet_manager.cc:1905] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:39.685699 13113 ts_tablet_manager.cc:1905] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:39.704156 13113 ts_tablet_manager.cc:1918] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:56:39.709951 13112 ts_tablet_manager.cc:1918] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:56:39.710254 13113 log.cc:1196] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/d87bc91e728f47dfb2f79c5f2021daa4
I20260807 02:56:39.710865 13113 ts_tablet_manager.cc:1939] T d87bc91e728f47dfb2f79c5f2021daa4 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting consensus metadata
I20260807 02:56:39.712608 12355 catalog_manager.cc:4841] TS aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255): tablet d87bc91e728f47dfb2f79c5f2021daa4 (table test-table [id=4364f0732ac24394b1ba88cfb173487b]) successfully deleted
I20260807 02:56:39.713128 13112 log.cc:1196] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/d87bc91e728f47dfb2f79c5f2021daa4
I20260807 02:56:39.717376 13112 ts_tablet_manager.cc:1939] T d87bc91e728f47dfb2f79c5f2021daa4 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting consensus metadata
I20260807 02:56:39.718811 12360 catalog_manager.cc:4841] TS 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693): tablet d87bc91e728f47dfb2f79c5f2021daa4 (table test-table [id=4364f0732ac24394b1ba88cfb173487b]) successfully deleted
I20260807 02:56:39.729348 13158 tablet_replica.cc:330] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: stopping tablet replica
I20260807 02:56:39.729710 13158 raft_consensus.cc:2229] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:56:39.730487 13158 raft_consensus.cc:2258] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:39.734279 13158 ts_tablet_manager.cc:1905] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:39.744051 13158 ts_tablet_manager.cc:1918] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:56:39.744302 13158 log.cc:1196] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/d87bc91e728f47dfb2f79c5f2021daa4
I20260807 02:56:39.744973 13158 ts_tablet_manager.cc:1939] T d87bc91e728f47dfb2f79c5f2021daa4 P 3b233536875345b19c467dd322f1f82e: Deleting consensus metadata
I20260807 02:56:39.746806 12361 catalog_manager.cc:4841] TS 3b233536875345b19c467dd322f1f82e (127.1.121.66:43617): tablet d87bc91e728f47dfb2f79c5f2021daa4 (table test-table [id=4364f0732ac24394b1ba88cfb173487b]) successfully deleted
Aug 07 02:56:39 dist-test-slave-3h5h krb5kdc[12313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:56:39 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:39 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:56:39.802837 13135 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:39.787461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.1.121.126:43931)
0807 02:56:39.787661 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.787668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.788216 (+ 548us) server_negotiation.cc:407] Connection header received
0807 02:56:39.788415 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.788423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.788555 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.788698 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:39.790507 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.791880 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.794004 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.794824 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.794996 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.802493 (+ 7497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34818: BlockingRecv error: recv got EOF from 127.0.0.1:34818 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 07 02:56:39 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM
I20260807 02:56:39.849503 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:39.829263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.1.121.126:43931)
0807 02:56:39.829604 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:39.829613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:39.829892 (+ 279us) server_negotiation.cc:407] Connection header received
0807 02:56:39.830108 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:39.830115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:39.830217 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:39.830357 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:39.832090 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.833458 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.835350 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:39.836150 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:39.836309 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:39.841677 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:39.841688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:39.841694 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:39.841747 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:39.845981 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:39.846823 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:39.846830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:39.846835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:39.847022 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:39.847468 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:39.847474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:39.847479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:39.847952 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:39.848254 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:39.848568 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 02:56:39.849035 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260807 02:56:39.854280 12376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:56:39.855201 12376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:56:39.863348 12761 tablet_service.cc:1467] Processing CreateTablet for tablet dd8899008bdd473684da2884222c2804 (DEFAULT_TABLE table=test-table [id=c49d01335af14da8b255826e3255f5b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:39.863653 12567 tablet_service.cc:1467] Processing CreateTablet for tablet dd8899008bdd473684da2884222c2804 (DEFAULT_TABLE table=test-table [id=c49d01335af14da8b255826e3255f5b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:39.863698 12952 tablet_service.cc:1467] Processing CreateTablet for tablet dd8899008bdd473684da2884222c2804 (DEFAULT_TABLE table=test-table [id=c49d01335af14da8b255826e3255f5b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:39.863963 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8899008bdd473684da2884222c2804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:39.864128 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8899008bdd473684da2884222c2804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:39.864154 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8899008bdd473684da2884222c2804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:39.871002 13085 tablet_bootstrap.cc:492] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: Bootstrap starting.
I20260807 02:56:39.871583 13087 tablet_bootstrap.cc:492] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: Bootstrap starting.
I20260807 02:56:39.872360 13086 tablet_bootstrap.cc:492] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: Bootstrap starting.
I20260807 02:56:39.872804 13085 tablet_bootstrap.cc:654] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:39.873356 13087 tablet_bootstrap.cc:654] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:39.874536 13086 tablet_bootstrap.cc:654] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:39.883862 13087 tablet_bootstrap.cc:492] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: No bootstrap required, opened a new log
I20260807 02:56:39.884088 13087 ts_tablet_manager.cc:1397] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260807 02:56:39.884300 13085 tablet_bootstrap.cc:492] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: No bootstrap required, opened a new log
I20260807 02:56:39.884529 13085 ts_tablet_manager.cc:1397] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260807 02:56:39.885053 13086 tablet_bootstrap.cc:492] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: No bootstrap required, opened a new log
I20260807 02:56:39.885042 13087 raft_consensus.cc:348] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.885315 13086 ts_tablet_manager.cc:1397] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260807 02:56:39.885461 13087 raft_consensus.cc:374] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:39.885418 13085 raft_consensus.cc:348] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.885634 13087 raft_consensus.cc:729] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aceefefa325e4b1da3c13b5c1b121e8c, State: Initialized, Role: FOLLOWER
I20260807 02:56:39.885777 13085 raft_consensus.cc:374] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:39.885951 13085 raft_consensus.cc:729] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d34b988fbe4c0691eb7f8c18e432a1, State: Initialized, Role: FOLLOWER
I20260807 02:56:39.886304 13085 consensus_queue.cc:260] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.886329 13086 raft_consensus.cc:348] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.887171 13085 ts_tablet_manager.cc:1428] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 02:56:39.887099 13087 consensus_queue.cc:260] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.887764 13086 raft_consensus.cc:374] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:39.887972 13086 raft_consensus.cc:729] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b233536875345b19c467dd322f1f82e, State: Initialized, Role: FOLLOWER
I20260807 02:56:39.888358 13086 consensus_queue.cc:260] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:39.890126 13086 ts_tablet_manager.cc:1428] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 02:56:39.891860 13087 ts_tablet_manager.cc:1428] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260807 02:56:40.046986 13095 raft_consensus.cc:482] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:40.047302 13095 raft_consensus.cc:504] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.049278 13095 leader_election.cc:288] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693), 3b233536875345b19c467dd322f1f82e (127.1.121.66:43617)
I20260807 02:56:40.059643 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:40.049774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:43617 (local address 127.1.121.67:49943)
0807 02:56:40.051570 (+ 1796us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:40.051593 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:40.051801 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:40.052373 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:40.052380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:40.052409 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:40.052903 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:40.052923 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.054180 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.054188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:40.056997 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:40.057015 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.059036 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.059041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:40.059168 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:40.059195 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 02:56:40.059264 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 02:56:40.059360 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1644,"mutex_wait_us":150,"thread_start_us":661,"threads_started":1}
I20260807 02:56:40.059824 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:40.049921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:49943 (local address 127.1.121.66:43617)
0807 02:56:40.050223 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:40.050232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:40.051691 (+ 1459us) server_negotiation.cc:407] Connection header received
0807 02:56:40.052009 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:40.052016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:40.052119 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:40.052248 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:40.053102 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.053972 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.057235 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.058802 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.059109 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:40.059215 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:40.059298 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 02:56:40.059481 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 02:56:40.060532 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8899008bdd473684da2884222c2804" candidate_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b233536875345b19c467dd322f1f82e" is_pre_election: true
I20260807 02:56:40.060873 12789 raft_consensus.cc:2454] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aceefefa325e4b1da3c13b5c1b121e8c in term 0.
I20260807 02:56:40.061527 12891 leader_election.cc:302] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b233536875345b19c467dd322f1f82e, aceefefa325e4b1da3c13b5c1b121e8c; no voters:
I20260807 02:56:40.062053 13095 raft_consensus.cc:2781] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:56:40.062217 13095 raft_consensus.cc:482] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:56:40.062345 13095 raft_consensus.cc:3037] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:40.063318 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:40.049487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:42693 (local address 127.1.121.67:54579)
0807 02:56:40.050721 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:40.050749 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:40.050939 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:40.052176 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:40.052181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:40.052207 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:56:40.052562 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:40.052576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.054378 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.054386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:40.056997 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:40.057015 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.061974 (+ 4959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.061980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:40.062527 (+ 547us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:40.062570 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 02:56:40.062723 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 02:56:40.062878 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":104,"thread_start_us":1756,"threads_started":1}
I20260807 02:56:40.063349 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:40.050266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:54579 (local address 127.1.121.65:42693)
0807 02:56:40.051844 (+ 1578us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:40.051853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:40.051877 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:56:40.051966 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:40.051970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:40.052034 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:40.052158 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:56:40.052797 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.054065 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.060151 (+ 6086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.061643 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.061942 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:40.062042 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:40.062759 (+ 717us) server_negotiation.cc:299] Negotiation successful
0807 02:56:40.062952 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1407,"thread_start_us":1295,"threads_started":1}
I20260807 02:56:40.064316 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8899008bdd473684da2884222c2804" candidate_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" is_pre_election: true
I20260807 02:56:40.064698 12596 raft_consensus.cc:2454] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aceefefa325e4b1da3c13b5c1b121e8c in term 0.
I20260807 02:56:40.065594 13095 raft_consensus.cc:504] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.066253 13095 leader_election.cc:288] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [CANDIDATE]: Term 1 election: Requested vote from peers 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693), 3b233536875345b19c467dd322f1f82e (127.1.121.66:43617)
I20260807 02:56:40.067013 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8899008bdd473684da2884222c2804" candidate_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d34b988fbe4c0691eb7f8c18e432a1"
I20260807 02:56:40.067039 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8899008bdd473684da2884222c2804" candidate_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b233536875345b19c467dd322f1f82e"
I20260807 02:56:40.067293 12789 raft_consensus.cc:3037] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:40.067271 12596 raft_consensus.cc:3037] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:40.069957 12789 raft_consensus.cc:2454] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aceefefa325e4b1da3c13b5c1b121e8c in term 1.
I20260807 02:56:40.070430 12891 leader_election.cc:302] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b233536875345b19c467dd322f1f82e, aceefefa325e4b1da3c13b5c1b121e8c; no voters:
I20260807 02:56:40.070770 13095 raft_consensus.cc:2781] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:40.071722 13095 raft_consensus.cc:686] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 LEADER]: Becoming Leader. State: Replica: aceefefa325e4b1da3c13b5c1b121e8c, State: Running, Role: LEADER
I20260807 02:56:40.072219 13095 consensus_queue.cc:237] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.071352 12596 raft_consensus.cc:2454] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aceefefa325e4b1da3c13b5c1b121e8c in term 1.
I20260807 02:56:40.075971 12374 catalog_manager.cc:5495] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c reported cstate change: term changed from 0 to 1, leader changed from <none> to aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67). New cstate: current_term: 1 leader_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } health_report { overall_health: HEALTHY } } }
I20260807 02:56:40.129640 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:40.121839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.1.121.67:43255)
0807 02:56:40.122607 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:40.122614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:40.122644 (+ 30us) server_negotiation.cc:407] Connection header received
0807 02:56:40.122774 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:40.122780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:40.122895 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:40.123100 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:56:40.123583 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.124922 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.126373 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.127134 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.127278 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:40.127938 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:56:40.128791 (+ 853us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:56:40.129013 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:40.129090 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 02:56:40.129186 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":367,"threads_started":1}
I20260807 02:56:40.133792 12596 raft_consensus.cc:1261] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Refusing update from remote peer aceefefa325e4b1da3c13b5c1b121e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:40.133792 12789 raft_consensus.cc:1261] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Refusing update from remote peer aceefefa325e4b1da3c13b5c1b121e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:40.134655 13185 consensus_queue.cc:1035] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:40.135123 13095 consensus_queue.cc:1035] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:40.155493 12374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:40.155813 12374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:40.160081 12374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238: Sending DeleteTablet for 3 replicas of tablet dd8899008bdd473684da2884222c2804
I20260807 02:56:40.161054 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8899008bdd473684da2884222c2804 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:40 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:50008
I20260807 02:56:40.161607 13112 tablet_replica.cc:330] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: stopping tablet replica
I20260807 02:56:40.161818 13112 raft_consensus.cc:2229] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:40.162231 13112 raft_consensus.cc:2258] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:40.162598 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8899008bdd473684da2884222c2804 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:40 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:35514
I20260807 02:56:40.162937 13158 tablet_replica.cc:330] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: stopping tablet replica
I20260807 02:56:40.163164 13158 raft_consensus.cc:2229] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:40.163180 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8899008bdd473684da2884222c2804 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:40 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:35858
I20260807 02:56:40.163406 13158 raft_consensus.cc:2258] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:40.163481 13113 tablet_replica.cc:330] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: stopping tablet replica
I20260807 02:56:40.163664 13113 raft_consensus.cc:2229] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:56:40.164901 13158 ts_tablet_manager.cc:1905] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:40.165973 13113 raft_consensus.cc:2258] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:40.167373 13112 ts_tablet_manager.cc:1905] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:40.169899 13113 ts_tablet_manager.cc:1905] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:40.172821 13158 ts_tablet_manager.cc:1918] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:40.173036 13158 log.cc:1196] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/dd8899008bdd473684da2884222c2804
I20260807 02:56:40.173540 13158 ts_tablet_manager.cc:1939] T dd8899008bdd473684da2884222c2804 P 3b233536875345b19c467dd322f1f82e: Deleting consensus metadata
I20260807 02:56:40.174906 12361 catalog_manager.cc:4841] TS 3b233536875345b19c467dd322f1f82e (127.1.121.66:43617): tablet dd8899008bdd473684da2884222c2804 (table test-table [id=c49d01335af14da8b255826e3255f5b0]) successfully deleted
I20260807 02:56:40.179136 13112 ts_tablet_manager.cc:1918] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:40.179358 13112 log.cc:1196] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/dd8899008bdd473684da2884222c2804
I20260807 02:56:40.179912 13112 ts_tablet_manager.cc:1939] T dd8899008bdd473684da2884222c2804 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting consensus metadata
I20260807 02:56:40.181252 12360 catalog_manager.cc:4841] TS 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693): tablet dd8899008bdd473684da2884222c2804 (table test-table [id=c49d01335af14da8b255826e3255f5b0]) successfully deleted
I20260807 02:56:40.181624 13113 ts_tablet_manager.cc:1918] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:56:40.181782 13113 log.cc:1196] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/dd8899008bdd473684da2884222c2804
I20260807 02:56:40.182294 13113 ts_tablet_manager.cc:1939] T dd8899008bdd473684da2884222c2804 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting consensus metadata
I20260807 02:56:40.183944 12355 catalog_manager.cc:4841] TS aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255): tablet dd8899008bdd473684da2884222c2804 (table test-table [id=c49d01335af14da8b255826e3255f5b0]) successfully deleted
I20260807 02:56:40.184195 12374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:56:40.185034 12374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:56:40.194168 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 080a059a01fc47a5bbcbab236adf5784 (DEFAULT_TABLE table=test-table [id=4eea66a0c04b413a857740454596c127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:40.194693 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080a059a01fc47a5bbcbab236adf5784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:40.195252 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 080a059a01fc47a5bbcbab236adf5784 (DEFAULT_TABLE table=test-table [id=4eea66a0c04b413a857740454596c127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:40.195734 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080a059a01fc47a5bbcbab236adf5784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:40.201350 12761 tablet_service.cc:1467] Processing CreateTablet for tablet 080a059a01fc47a5bbcbab236adf5784 (DEFAULT_TABLE table=test-table [id=4eea66a0c04b413a857740454596c127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:56:40.201798 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080a059a01fc47a5bbcbab236adf5784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:40.201943 13085 tablet_bootstrap.cc:492] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: Bootstrap starting.
I20260807 02:56:40.203687 13085 tablet_bootstrap.cc:654] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:40.211794 13086 tablet_bootstrap.cc:492] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: Bootstrap starting.
I20260807 02:56:40.213652 13086 tablet_bootstrap.cc:654] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:40.214747 13085 tablet_bootstrap.cc:492] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: No bootstrap required, opened a new log
I20260807 02:56:40.214926 13085 ts_tablet_manager.cc:1397] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260807 02:56:40.215734 13085 raft_consensus.cc:348] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.216022 13085 raft_consensus.cc:374] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:40.216113 13085 raft_consensus.cc:729] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d34b988fbe4c0691eb7f8c18e432a1, State: Initialized, Role: FOLLOWER
I20260807 02:56:40.216408 13085 consensus_queue.cc:260] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.217074 13087 tablet_bootstrap.cc:492] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: Bootstrap starting.
I20260807 02:56:40.218772 13087 tablet_bootstrap.cc:654] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:40.222518 13085 ts_tablet_manager.cc:1428] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260807 02:56:40.238085 13087 tablet_bootstrap.cc:492] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: No bootstrap required, opened a new log
I20260807 02:56:40.238270 13087 ts_tablet_manager.cc:1397] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20260807 02:56:40.239143 13087 raft_consensus.cc:348] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.239455 13087 raft_consensus.cc:374] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:40.239560 13087 raft_consensus.cc:729] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aceefefa325e4b1da3c13b5c1b121e8c, State: Initialized, Role: FOLLOWER
I20260807 02:56:40.239817 13087 consensus_queue.cc:260] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.242717 13087 ts_tablet_manager.cc:1428] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260807 02:56:40.244031 13086 tablet_bootstrap.cc:492] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: No bootstrap required, opened a new log
I20260807 02:56:40.244339 13086 ts_tablet_manager.cc:1397] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.008s
I20260807 02:56:40.245612 13086 raft_consensus.cc:348] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.246091 13086 raft_consensus.cc:374] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:40.246261 13086 raft_consensus.cc:729] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b233536875345b19c467dd322f1f82e, State: Initialized, Role: FOLLOWER
I20260807 02:56:40.246673 13086 consensus_queue.cc:260] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.250551 13086 ts_tablet_manager.cc:1428] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260807 02:56:40.455024 13123 raft_consensus.cc:482] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:56:40.455322 13123 raft_consensus.cc:504] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.456271 13123 leader_election.cc:288] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693), aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255)
I20260807 02:56:40.457561 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080a059a01fc47a5bbcbab236adf5784" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" is_pre_election: true
I20260807 02:56:40.457865 12596 raft_consensus.cc:2454] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 0.
I20260807 02:56:40.458618 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080a059a01fc47a5bbcbab236adf5784" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" is_pre_election: true
I20260807 02:56:40.459000 12988 raft_consensus.cc:2454] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 0.
I20260807 02:56:40.459610 12699 leader_election.cc:302] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d34b988fbe4c0691eb7f8c18e432a1, 3b233536875345b19c467dd322f1f82e; no voters:
I20260807 02:56:40.460364 13123 raft_consensus.cc:2781] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:56:40.460525 13123 raft_consensus.cc:482] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:56:40.460683 13123 raft_consensus.cc:3037] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:40.467633 13123 raft_consensus.cc:504] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.468912 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080a059a01fc47a5bbcbab236adf5784" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d34b988fbe4c0691eb7f8c18e432a1"
I20260807 02:56:40.469168 12596 raft_consensus.cc:3037] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:40.472338 12596 raft_consensus.cc:2454] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 1.
I20260807 02:56:40.472944 12699 leader_election.cc:302] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d34b988fbe4c0691eb7f8c18e432a1, 3b233536875345b19c467dd322f1f82e; no voters:
I20260807 02:56:40.481819 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080a059a01fc47a5bbcbab236adf5784" candidate_uuid: "3b233536875345b19c467dd322f1f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aceefefa325e4b1da3c13b5c1b121e8c"
I20260807 02:56:40.481927 13199 raft_consensus.cc:2781] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:40.482131 12988 raft_consensus.cc:3037] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:40.482138 13199 raft_consensus.cc:686] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 1 LEADER]: Becoming Leader. State: Replica: 3b233536875345b19c467dd322f1f82e, State: Running, Role: LEADER
I20260807 02:56:40.482514 13199 consensus_queue.cc:237] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } }
I20260807 02:56:40.485844 12988 raft_consensus.cc:2454] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b233536875345b19c467dd322f1f82e in term 1.
I20260807 02:56:40.486654 13123 leader_election.cc:288] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [CANDIDATE]: Term 1 election: Requested vote from peers 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693), aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255)
I20260807 02:56:40.491034 12375 catalog_manager.cc:5495] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b233536875345b19c467dd322f1f82e (127.1.121.66). New cstate: current_term: 1 leader_uuid: "3b233536875345b19c467dd322f1f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b233536875345b19c467dd322f1f82e" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 43617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 } health_report { overall_health: UNKNOWN } } }
I20260807 02:56:40.550846 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:40.542777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.1.121.66:43617)
0807 02:56:40.543040 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:40.543047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:40.543090 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:56:40.543166 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:40.543171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:40.543317 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:40.543420 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:56:40.543957 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.545388 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.547748 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.548570 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.548738 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:40.549495 (+ 757us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:56:40.549677 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:56:40.549774 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:40.550373 (+ 599us) server_negotiation.cc:299] Negotiation successful
0807 02:56:40.550474 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260807 02:56:40.569346 12596 raft_consensus.cc:1261] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Refusing update from remote peer 3b233536875345b19c467dd322f1f82e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:40.570014 13200 consensus_queue.cc:1035] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d34b988fbe4c0691eb7f8c18e432a1" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:40.570777 12988 raft_consensus.cc:1261] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Refusing update from remote peer 3b233536875345b19c467dd322f1f82e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:56:40.571460 13200 consensus_queue.cc:1035] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aceefefa325e4b1da3c13b5c1b121e8c" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:56:40.668223 12374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:40.668497 12374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:56:40.672564 12374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8989c159ac64a4dba7df8841fc7b238: Sending DeleteTablet for 3 replicas of tablet 080a059a01fc47a5bbcbab236adf5784
I20260807 02:56:40.673553 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 080a059a01fc47a5bbcbab236adf5784 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:40 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:50008
I20260807 02:56:40.673866 13112 tablet_replica.cc:330] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: stopping tablet replica
I20260807 02:56:40.673934 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 080a059a01fc47a5bbcbab236adf5784 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:40 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:35514
I20260807 02:56:40.674068 13112 raft_consensus.cc:2229] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:40.674252 13112 raft_consensus.cc:2258] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:40.674263 13158 tablet_replica.cc:330] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: stopping tablet replica
I20260807 02:56:40.674450 13158 raft_consensus.cc:2229] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:56:40.674873 13158 raft_consensus.cc:2258] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:40.675477 13112 ts_tablet_manager.cc:1905] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:40.676152 13158 ts_tablet_manager.cc:1905] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:40.681768 12952 tablet_service.cc:1514] Processing DeleteTablet for tablet 080a059a01fc47a5bbcbab236adf5784 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:56:40 UTC) from {username='oryx', principal='oryx/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:35858
I20260807 02:56:40.682080 13113 tablet_replica.cc:330] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: stopping tablet replica
I20260807 02:56:40.682287 13113 raft_consensus.cc:2229] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:56:40.682354 13112 ts_tablet_manager.cc:1918] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:56:40.682502 13112 log.cc:1196] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/080a059a01fc47a5bbcbab236adf5784
I20260807 02:56:40.682489 13113 raft_consensus.cc:2258] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:56:40.682884 13112 ts_tablet_manager.cc:1939] T 080a059a01fc47a5bbcbab236adf5784 P 18d34b988fbe4c0691eb7f8c18e432a1: Deleting consensus metadata
I20260807 02:56:40.684049 13113 ts_tablet_manager.cc:1905] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:56:40.684062 12360 catalog_manager.cc:4841] TS 18d34b988fbe4c0691eb7f8c18e432a1 (127.1.121.65:42693): tablet 080a059a01fc47a5bbcbab236adf5784 (table test-table [id=4eea66a0c04b413a857740454596c127]) successfully deleted
I20260807 02:56:40.688162 13158 ts_tablet_manager.cc:1918] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:56:40.688297 13158 log.cc:1196] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/080a059a01fc47a5bbcbab236adf5784
I20260807 02:56:40.688789 13158 ts_tablet_manager.cc:1939] T 080a059a01fc47a5bbcbab236adf5784 P 3b233536875345b19c467dd322f1f82e: Deleting consensus metadata
I20260807 02:56:40.689955 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 12469
I20260807 02:56:40.691938 12361 catalog_manager.cc:4841] TS 3b233536875345b19c467dd322f1f82e (127.1.121.66:43617): tablet 080a059a01fc47a5bbcbab236adf5784 (table test-table [id=4eea66a0c04b413a857740454596c127]) successfully deleted
I20260807 02:56:40.692791 13113 ts_tablet_manager.cc:1918] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:56:40.692974 13113 log.cc:1196] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/080a059a01fc47a5bbcbab236adf5784
I20260807 02:56:40.693403 13113 ts_tablet_manager.cc:1939] T 080a059a01fc47a5bbcbab236adf5784 P aceefefa325e4b1da3c13b5c1b121e8c: Deleting consensus metadata
I20260807 02:56:40.694787 12355 catalog_manager.cc:4841] TS aceefefa325e4b1da3c13b5c1b121e8c (127.1.121.67:43255): tablet 080a059a01fc47a5bbcbab236adf5784 (table test-table [id=4eea66a0c04b413a857740454596c127]) successfully deleted
Aug 07 02:56:40 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.65@KRBTEST.COM
I20260807 02:56:40.716708 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:40.691651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.1.121.65:42693)
0807 02:56:40.695448 (+ 3797us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:40.695455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:40.695920 (+ 465us) server_negotiation.cc:407] Connection header received
0807 02:56:40.696090 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:40.696097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:40.696269 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:40.696388 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:40.698059 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.699378 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.700736 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.701492 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.701640 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:40.709649 (+ 8009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:40.709672 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:40.709695 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:40.709744 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:40.714074 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:40.714805 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:40.714814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:40.714816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:40.714950 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:40.715291 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:40.715294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:40.715296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:40.715703 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:40.715963 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:40.716208 (+ 245us) server_negotiation.cc:299] Negotiation successful
0807 02:56:40.716391 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3654,"thread_start_us":3535,"threads_started":1}
I20260807 02:56:40.717489 12653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:40.861461 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 12469
I20260807 02:56:40.894093 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 12667
Aug 07 02:56:40 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.66@KRBTEST.COM
I20260807 02:56:40.909582 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:40.895061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.1.121.66:43617)
0807 02:56:40.895265 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:40.895269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:40.895286 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:56:40.895344 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:40.895347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:40.895407 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:40.895490 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:40.896759 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.897579 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.898853 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:40.899293 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:40.899396 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:40.903938 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:40.903953 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:40.903972 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:40.904021 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:40.907061 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:40.907641 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:40.907648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:40.907651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:40.907794 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:40.908169 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:40.908172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:40.908174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:40.908534 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:40.908774 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:40.909141 (+ 367us) server_negotiation.cc:299] Negotiation successful
0807 02:56:40.909309 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:56:40.910250 12852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:41.062155 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 12667
W20260807 02:56:41.101069 12891 connection.cc:535] server connection from 127.1.121.66:46863 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 02:56:41.101348 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 12866
Aug 07 02:56:41 dist-test-slave-3h5h krb5kdc[12313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.121.67@KRBTEST.COM
I20260807 02:56:41.117080 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:41.102116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.1.121.67:43255)
0807 02:56:41.102653 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:41.102658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:41.102675 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:56:41.102732 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:41.102735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:41.102802 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:41.102913 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:41.104281 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:41.105079 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:41.106268 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:41.106689 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:41.106790 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:41.111174 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:41.111190 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:41.111208 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:41.111243 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:41.114454 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:41.115091 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:41.115099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:41.115101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:41.115233 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:41.115575 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:41.115579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:41.115581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:41.115953 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:41.116198 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:41.116445 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 02:56:41.116638 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":308,"threads_started":1}
I20260807 02:56:41.117873 13048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:41.245004 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 12866
I20260807 02:56:41.278863 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 12331
I20260807 02:56:41.280220 12431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:41.401366 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 12331
2026-08-07T02:56:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (8008 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:56:41 dist-test-slave-3h5h krb5kdc[13222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:56:41 dist-test-slave-3h5h krb5kdc[13222](info): set up 2 sockets
Aug 07 02:56:41 dist-test-slave-3h5h krb5kdc[13222](info): commencing operation
krb5kdc: starting...
W20260807 02:56:42.814798 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.357s user 0.006s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:56:43 dist-test-slave-3h5h krb5kdc[13222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:56:43.130415 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37415
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:43.483448 13235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:43.483799 13235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:43.483973 13235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:43.491279 13235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:43.491427 13235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:43.491534 13235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:43.491642 13235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:43.491734 13235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:43.501677 13235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:37415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:43.503080 13235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:43.504410 13235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:43.516713 13244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:43.516863 13241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:43.516901 13242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:43.517935 13235 server_base.cc:1034] running on GCE node
I20260807 02:56:43.518847 13235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:43.520390 13235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:43.521605 13235 hybrid_clock.cc:648] HybridClock initialized: now 1786071403521551 us; error 38 us; skew 500 ppm
I20260807 02:56:43.522053 13235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T02:56:43Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2185 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:56:43 dist-test-slave-3h5h krb5kdc[13251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:56:43 dist-test-slave-3h5h krb5kdc[13251](info): set up 2 sockets
Aug 07 02:56:43 dist-test-slave-3h5h krb5kdc[13251](info): commencing operation
krb5kdc: starting...
W20260807 02:56:44.906860 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:56:45 dist-test-slave-3h5h krb5kdc[13251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:56:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:56:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:56:45.213361 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45727
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:45727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:45.565596 13265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:45.565882 13265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:45.566094 13265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:45.573201 13265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:45.573307 13265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:45.573444 13265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:45.573545 13265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:45.573633 13265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:45.583472 13265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:45727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:45.584899 13265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:45.586292 13265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:45.599120 13271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:45.599066 13272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:45.599045 13274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:45.600077 13265 server_base.cc:1034] running on GCE node
I20260807 02:56:45.600903 13265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:45.602383 13265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:45.603596 13265 hybrid_clock.cc:648] HybridClock initialized: now 1786071405603549 us; error 37 us; skew 500 ppm
I20260807 02:56:45.603960 13265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:45 dist-test-slave-3h5h krb5kdc[13251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071405, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:45.607339 13265 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:56:45.609045 13265 webserver.cc:460] Webserver started at http://127.1.121.126:45221/ using document root <none> and password file <none>
I20260807 02:56:45.609778 13265 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:45.609926 13265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:45.610267 13265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:45.613380 13265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef"
format_stamp: "Formatted at 2026-08-07 02:56:45 on dist-test-slave-3h5h"
server_key: "06b42ef8220c1c3bd539bc42c4c2552c"
server_key_iv: "4e03e0749285de4e48f6684b94096ecd"
server_key_version: "encryptionkey@0"
I20260807 02:56:45.614212 13265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef"
format_stamp: "Formatted at 2026-08-07 02:56:45 on dist-test-slave-3h5h"
server_key: "06b42ef8220c1c3bd539bc42c4c2552c"
server_key_iv: "4e03e0749285de4e48f6684b94096ecd"
server_key_version: "encryptionkey@0"
I20260807 02:56:45.619719 13265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 02:56:45.623553 13284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:45.625103 13265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:45.625308 13265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef"
format_stamp: "Formatted at 2026-08-07 02:56:45 on dist-test-slave-3h5h"
server_key: "06b42ef8220c1c3bd539bc42c4c2552c"
server_key_iv: "4e03e0749285de4e48f6684b94096ecd"
server_key_version: "encryptionkey@0"
I20260807 02:56:45.625522 13265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:45.640327 13265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:45.645160 13265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:45.645483 13265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:45.665189 13265 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:45727
I20260807 02:56:45.665201 13358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:45727 every 8 connection(s)
I20260807 02:56:45.666682 13265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:45.670745 13359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:56:45.673401 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 13265
I20260807 02:56:45.673633 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:56:45.674343 1509 external_mini_cluster.cc:1428] Setting key 2c9e04d208263611ff139668eee87f06
I20260807 02:56:45.684067 13359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: Bootstrap starting.
I20260807 02:56:45.687749 13359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: Neither blocks nor log segments found. Creating new log.
I20260807 02:56:45.688763 13359 log.cc:824] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:56:45 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:45.691713 13359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: No bootstrap required, opened a new log
I20260807 02:56:45.698930 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:45.676455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.1.121.126:45727)
0807 02:56:45.677343 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:45.677359 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:45.677411 (+ 52us) server_negotiation.cc:407] Connection header received
0807 02:56:45.679843 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:45.679877 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:45.680414 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:45.680895 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:45.682399 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:45.684374 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:45.685736 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:45.686782 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:45.686970 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:45.691341 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:45.691349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:45.691363 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:45.691407 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:45.694680 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:45.695434 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:45.695441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:45.695447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:45.695633 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:45.696048 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:45.696058 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:45.696060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:45.696629 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:45.696993 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:45.697453 (+ 460us) server_negotiation.cc:299] Negotiation successful
0807 02:56:45.697885 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":362,"threads_started":1}
I20260807 02:56:45.699755 13359 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } }
I20260807 02:56:45.700121 13359 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:56:45.700251 13359 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc06c1c601f14d9c8f8d5edf31b4b6ef, State: Initialized, Role: FOLLOWER
I20260807 02:56:45.700827 13359 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } }
I20260807 02:56:45.701159 13359 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:45.701326 13359 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:45.701520 13359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:56:45.703879 13359 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } }
I20260807 02:56:45.704388 13359 leader_election.cc:302] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc06c1c601f14d9c8f8d5edf31b4b6ef; no voters:
I20260807 02:56:45.705106 13359 leader_election.cc:288] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:56:45.705309 13367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:56:45.706303 13367 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 1 LEADER]: Becoming Leader. State: Replica: cc06c1c601f14d9c8f8d5edf31b4b6ef, State: Running, Role: LEADER
I20260807 02:56:45.706977 13367 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } }
I20260807 02:56:45.707419 13359 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:45.714658 13368 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } } }
I20260807 02:56:45.715227 13368 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:45.716198 13369 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc06c1c601f14d9c8f8d5edf31b4b6ef. Latest consensus state: current_term: 1 leader_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } } }
I20260807 02:56:45.716579 13369 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: This master's current role is: LEADER
W20260807 02:56:45.721076 13387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:56:45.721307 13387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:56:45.721505 13388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:45.723282 13388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:45.733189 13388 catalog_manager.cc:1347] Generated new cluster ID: d3f7b72a2b6f4e669d7541f48f7f5d1e
I20260807 02:56:45.733343 13388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:45.762300 13388 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:56:45.763870 13388 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:45.773221 13388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: Generated new TSK 0
I20260807 02:56:45.774268 13388 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:56:45.854183 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45727
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:46.220513 13397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:46.220817 13397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:46.221033 13397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:46.228524 13397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:46.228639 13397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:46.228880 13397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:56:46.239006 13397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:46.240356 13397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:46.241818 13397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:46.255390 13406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:46.255390 13407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:46.255638 13410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:46.256469 13397 server_base.cc:1034] running on GCE node
I20260807 02:56:46.257191 13397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:46.258155 13397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:46.259392 13397 hybrid_clock.cc:648] HybridClock initialized: now 1786071406259355 us; error 53 us; skew 500 ppm
I20260807 02:56:46.259714 13397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:46 dist-test-slave-3h5h krb5kdc[13251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071406, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:46.269135 13397 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:56:46.270692 13397 webserver.cc:460] Webserver started at http://127.1.121.65:32875/ using document root <none> and password file <none>
I20260807 02:56:46.271445 13397 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:46.271596 13397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:46.271917 13397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:46.275130 13397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "91ba6ee7f7b54cc991f0955cc1eb7a34"
format_stamp: "Formatted at 2026-08-07 02:56:46 on dist-test-slave-3h5h"
server_key: "66871f675ef88c147866358cf73129dc"
server_key_iv: "dba44d2a329d7dd60867407153e7a86d"
server_key_version: "encryptionkey@0"
I20260807 02:56:46.275943 13397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "91ba6ee7f7b54cc991f0955cc1eb7a34"
format_stamp: "Formatted at 2026-08-07 02:56:46 on dist-test-slave-3h5h"
server_key: "66871f675ef88c147866358cf73129dc"
server_key_iv: "dba44d2a329d7dd60867407153e7a86d"
server_key_version: "encryptionkey@0"
I20260807 02:56:46.281286 13397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 02:56:46.284889 13419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:46.286237 13397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:56:46.286422 13397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "91ba6ee7f7b54cc991f0955cc1eb7a34"
format_stamp: "Formatted at 2026-08-07 02:56:46 on dist-test-slave-3h5h"
server_key: "66871f675ef88c147866358cf73129dc"
server_key_iv: "dba44d2a329d7dd60867407153e7a86d"
server_key_version: "encryptionkey@0"
I20260807 02:56:46.286628 13397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:46.320276 13397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:46.325489 13397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:46.325842 13397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:46.326984 13397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:46.328495 13397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:46.328621 13397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:46.328835 13397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:46.328955 13397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 02:56:46.366521 13397 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:38017
I20260807 02:56:46.366557 13567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:38017 every 8 connection(s)
I20260807 02:56:46.367970 13397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:56:46.377048 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 13397
I20260807 02:56:46.377337 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:56:46.378119 1509 external_mini_cluster.cc:1428] Setting key 4cad354d74d2a63e524c1fa6dd1b03f6
Aug 07 02:56:46 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071406, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:46.389402 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:46.370814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:33311 (local address 127.1.121.126:45727)
0807 02:56:46.371317 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:46.371322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:46.372326 (+ 1004us) server_negotiation.cc:407] Connection header received
0807 02:56:46.374390 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:46.374396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:46.374491 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:46.374615 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:46.376118 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:46.376926 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:46.378661 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:46.379224 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:46.379371 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:46.383468 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:46.383473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:46.383476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:46.383512 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:46.386249 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:46.387009 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:46.387017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:46.387020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:46.387158 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:46.387591 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:46.387599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:46.387604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:46.387878 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:46.388039 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:46.388756 (+ 717us) server_negotiation.cc:299] Negotiation successful
0807 02:56:46.389007 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260807 02:56:46.390583 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:46.371419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:33311)
0807 02:56:46.372154 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:46.372197 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:46.374102 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:46.374860 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:46.374868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:46.375381 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:46.375943 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:46.375959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:46.377100 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:46.377104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:46.378506 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:46.378519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:46.379475 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:46.379485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:46.379735 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:46.380628 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:46.380652 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:46.383225 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:46.386399 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:46.386409 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:46.386425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:46.386874 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:46.387274 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:46.387279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:46.387283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:46.387475 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:46.388015 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:46.388024 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:46.388367 (+ 343us) client_negotiation.cc:769] Sending connection context
0807 02:56:46.388656 (+ 289us) client_negotiation.cc:240] Negotiation successful
0807 02:56:46.389212 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":314,"threads_started":1}
I20260807 02:56:46.392308 13568 heartbeater.cc:347] Connected to a master server at 127.1.121.126:45727
I20260807 02:56:46.392633 13568 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:46.393440 13568 heartbeater.cc:511] Master 127.1.121.126:45727 requested a full tablet report, sending...
I20260807 02:56:46.395702 13308 ts_manager.cc:194] Registered new tserver with Master: 91ba6ee7f7b54cc991f0955cc1eb7a34 (127.1.121.65:38017)
I20260807 02:56:46.397235 13308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:33311
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:56:46.445235 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45727
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:46.799420 13579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:46.799710 13579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:46.799805 13579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:46.807207 13579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:46.807318 13579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:46.807435 13579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:56:46.817548 13579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:46.818876 13579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:46.820209 13579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:46.833997 13585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:46.834084 13584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:46.834088 13587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:46.835839 13579 server_base.cc:1034] running on GCE node
I20260807 02:56:46.836519 13579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:46.837566 13579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:46.838775 13579 hybrid_clock.cc:648] HybridClock initialized: now 1786071406838707 us; error 59 us; skew 500 ppm
I20260807 02:56:46.839089 13579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:46 dist-test-slave-3h5h krb5kdc[13251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071406, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:46.842491 13579 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:56:46.844163 13579 webserver.cc:460] Webserver started at http://127.1.121.66:37591/ using document root <none> and password file <none>
I20260807 02:56:46.844996 13579 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:46.845155 13579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:46.845497 13579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:46.848515 13579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "b1e04d475710407a9c3c7b7642542d73"
format_stamp: "Formatted at 2026-08-07 02:56:46 on dist-test-slave-3h5h"
server_key: "8e2d23817fa89c9dcce8f1b3c124d338"
server_key_iv: "4100fc3c60db97800aa4ca08605f11a2"
server_key_version: "encryptionkey@0"
I20260807 02:56:46.849386 13579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "b1e04d475710407a9c3c7b7642542d73"
format_stamp: "Formatted at 2026-08-07 02:56:46 on dist-test-slave-3h5h"
server_key: "8e2d23817fa89c9dcce8f1b3c124d338"
server_key_iv: "4100fc3c60db97800aa4ca08605f11a2"
server_key_version: "encryptionkey@0"
I20260807 02:56:46.854771 13579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 02:56:46.858632 13596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:46.860031 13579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:56:46.860234 13579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "b1e04d475710407a9c3c7b7642542d73"
format_stamp: "Formatted at 2026-08-07 02:56:46 on dist-test-slave-3h5h"
server_key: "8e2d23817fa89c9dcce8f1b3c124d338"
server_key_iv: "4100fc3c60db97800aa4ca08605f11a2"
server_key_version: "encryptionkey@0"
I20260807 02:56:46.860448 13579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:46.886744 13579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:46.891562 13579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:46.891872 13579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:46.892913 13579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:46.894413 13579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:46.894524 13579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:46.894681 13579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:46.894757 13579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:46.930902 13579 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:42285
I20260807 02:56:46.930998 13740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:42285 every 8 connection(s)
I20260807 02:56:46.932289 13579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:56:46.935460 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 13579
I20260807 02:56:46.935657 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:56:46.936151 1509 external_mini_cluster.cc:1428] Setting key a40709ab5582b6b7e6c2db99eb0ef912
Aug 07 02:56:46 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071406, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:46.956236 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:46.935333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:57137 (local address 127.1.121.126:45727)
0807 02:56:46.935886 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:46.935893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:46.937251 (+ 1358us) server_negotiation.cc:407] Connection header received
0807 02:56:46.939566 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:46.939576 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:46.939668 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:46.939803 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:46.941579 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:46.942702 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:46.944943 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:46.945703 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:46.945852 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:46.949679 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:46.949688 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:46.949696 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:46.949744 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:46.952734 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:46.953733 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:46.953737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:46.953739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:46.953860 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:46.954420 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:46.954424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:46.954426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:46.954715 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:46.954828 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:46.955784 (+ 956us) server_negotiation.cc:299] Negotiation successful
0807 02:56:46.955964 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":278,"threads_started":1}
I20260807 02:56:46.957898 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:46.936031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:57137)
0807 02:56:46.937057 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:46.937109 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:46.939254 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:46.940051 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:46.940060 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:46.940686 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:46.941359 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:46.941377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:46.942866 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:46.942874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:46.944755 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:46.944777 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:46.945927 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:46.945932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:46.946145 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:46.946865 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:46.946888 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:46.949419 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:46.952905 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:46.952917 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:46.952938 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:46.953594 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:46.953973 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:46.953981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:46.953987 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:46.954295 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:46.954850 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:46.954861 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:46.955337 (+ 476us) client_negotiation.cc:769] Sending connection context
0807 02:56:46.955661 (+ 324us) client_negotiation.cc:240] Negotiation successful
0807 02:56:46.956348 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":423,"threads_started":1}
I20260807 02:56:46.959578 13742 heartbeater.cc:347] Connected to a master server at 127.1.121.126:45727
I20260807 02:56:46.959950 13742 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:46.960773 13742 heartbeater.cc:511] Master 127.1.121.126:45727 requested a full tablet report, sending...
I20260807 02:56:46.962533 13308 ts_manager.cc:194] Registered new tserver with Master: b1e04d475710407a9c3c7b7642542d73 (127.1.121.66:42285)
I20260807 02:56:46.963424 13308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:57137
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:56:47.005326 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45727
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:47.378715 13750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:47.379051 13750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:47.379231 13750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:47.386821 13750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:47.386977 13750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:47.387171 13750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:56:47.397578 13750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:47.399003 13750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:47.400211 13568 heartbeater.cc:503] Master 127.1.121.126:45727 was elected leader, sending a full tablet report...
I20260807 02:56:47.400445 13750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:47.415616 13760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:47.415570 13758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:47.415642 13757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:47.416911 13750 server_base.cc:1034] running on GCE node
I20260807 02:56:47.417564 13750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:47.418581 13750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:47.419817 13750 hybrid_clock.cc:648] HybridClock initialized: now 1786071407419768 us; error 35 us; skew 500 ppm
I20260807 02:56:47.420190 13750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:47 dist-test-slave-3h5h krb5kdc[13251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:47.423485 13750 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:56:47.425410 13750 webserver.cc:460] Webserver started at http://127.1.121.67:37601/ using document root <none> and password file <none>
I20260807 02:56:47.426220 13750 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:47.426378 13750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:47.426721 13750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:56:47.430195 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "1a5d8e52b6ca4577899b7980edf6dacd"
format_stamp: "Formatted at 2026-08-07 02:56:47 on dist-test-slave-3h5h"
server_key: "06889c38bc571f9928b28e9de3ab6458"
server_key_iv: "5b55052f6d6c18f1449741c6ec46ee03"
server_key_version: "encryptionkey@0"
I20260807 02:56:47.431049 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "1a5d8e52b6ca4577899b7980edf6dacd"
format_stamp: "Formatted at 2026-08-07 02:56:47 on dist-test-slave-3h5h"
server_key: "06889c38bc571f9928b28e9de3ab6458"
server_key_iv: "5b55052f6d6c18f1449741c6ec46ee03"
server_key_version: "encryptionkey@0"
I20260807 02:56:47.436590 13750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 02:56:47.440469 13771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:47.442102 13750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:47.442298 13750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "1a5d8e52b6ca4577899b7980edf6dacd"
format_stamp: "Formatted at 2026-08-07 02:56:47 on dist-test-slave-3h5h"
server_key: "06889c38bc571f9928b28e9de3ab6458"
server_key_iv: "5b55052f6d6c18f1449741c6ec46ee03"
server_key_version: "encryptionkey@0"
I20260807 02:56:47.442521 13750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:47.470320 13750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:47.476269 13750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:47.476605 13750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:47.478093 13750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:47.481501 13750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:47.481611 13750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:47.481750 13750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:47.481822 13750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:47.524721 13750 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:37929
I20260807 02:56:47.524817 13909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:37929 every 8 connection(s)
I20260807 02:56:47.526134 13750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:56:47.527696 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 13750
I20260807 02:56:47.527878 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:56:47.528434 1509 external_mini_cluster.cc:1428] Setting key 2ca2b612967d35b30298a4b7c9814e72
Aug 07 02:56:47 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:56:47.547401 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:47.529213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:44257 (local address 127.1.121.126:45727)
0807 02:56:47.529787 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:47.529794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:47.531024 (+ 1230us) server_negotiation.cc:407] Connection header received
0807 02:56:47.533187 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:47.533191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:47.533260 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:47.533360 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:47.534915 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:47.535707 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:47.537439 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:47.537849 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:47.537949 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:47.541394 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:47.541399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:47.541402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:47.541436 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:47.544174 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:47.544955 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:47.544962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:47.544965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:47.545118 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:47.545585 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:47.545597 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:47.545599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:47.545892 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:47.546073 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:47.546780 (+ 707us) server_negotiation.cc:299] Negotiation successful
0807 02:56:47.547003 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":440,"threads_started":1}
I20260807 02:56:47.548333 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:47.529856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:44257)
0807 02:56:47.530831 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:47.530883 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:47.532930 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:47.533594 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:47.533603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:47.534137 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:47.534720 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:47.534738 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:47.535868 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:47.535873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:47.537301 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:47.537314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:47.538040 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:47.538045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:47.538240 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:47.538895 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:47.538917 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:47.541188 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:56:47.544334 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:47.544343 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:47.544359 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:47.544798 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:47.545238 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:56:47.545244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:56:47.545252 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:56:47.545455 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:56:47.546028 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:56:47.546036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:56:47.546400 (+ 364us) client_negotiation.cc:769] Sending connection context
0807 02:56:47.546673 (+ 273us) client_negotiation.cc:240] Negotiation successful
0807 02:56:47.547189 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":379,"threads_started":1}
I20260807 02:56:47.549793 13910 heartbeater.cc:347] Connected to a master server at 127.1.121.126:45727
I20260807 02:56:47.550130 13910 heartbeater.cc:464] Registering TS with master...
I20260807 02:56:47.550844 13910 heartbeater.cc:511] Master 127.1.121.126:45727 requested a full tablet report, sending...
I20260807 02:56:47.552309 13308 ts_manager.cc:194] Registered new tserver with Master: 1a5d8e52b6ca4577899b7980edf6dacd (127.1.121.67:37929)
I20260807 02:56:47.553197 13308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:44257
I20260807 02:56:47.557042 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab.
Entry for principal oryx/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab.
WARNING: no policy specified for oryx/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab.
Entry for principal oryx/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab.
WARNING: no policy specified for oryx/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab.
Entry for principal oryx/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab.
I20260807 02:56:47.760640 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 13397
Aug 07 02:56:47 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:56:47.779428 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:47.762052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.1.121.65:38017)
0807 02:56:47.762832 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:47.762840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:47.762868 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:56:47.762934 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:47.762941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:47.763245 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:47.763384 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:47.764915 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:47.766071 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:47.767500 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:47.768157 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:47.768302 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:47.772532 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:47.772545 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:47.772561 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:47.772599 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:47.776209 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:47.777116 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:47.777127 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:47.777130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:47.777323 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:47.777790 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:47.777796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:47.777799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:47.778286 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:47.778621 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:47.778920 (+ 299us) server_negotiation.cc:299] Negotiation successful
0807 02:56:47.779134 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":447,"threads_started":1}
I20260807 02:56:47.780803 13561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:47.865141 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 13397
I20260807 02:56:47.892388 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 13579
Aug 07 02:56:47 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:56:47.909832 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:47.893595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.1.121.66:42285)
0807 02:56:47.894190 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:47.894197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:47.894225 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:56:47.894288 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:47.894293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:47.894516 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:47.894664 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:47.896168 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:47.897400 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:47.898714 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:47.899270 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:47.899368 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:47.903973 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:47.903985 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:47.904001 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:47.904036 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:47.907009 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:47.907671 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:47.907677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:47.907680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:47.907883 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:47.908235 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:47.908238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:47.908240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:47.908585 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:47.908824 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:47.909211 (+ 387us) server_negotiation.cc:299] Negotiation successful
0807 02:56:47.909410 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":318,"threads_started":1}
I20260807 02:56:47.911355 13735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:47.996770 13742 heartbeater.cc:503] Master 127.1.121.126:45727 was elected leader, sending a full tablet report...
I20260807 02:56:47.998265 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 13579
W20260807 02:56:48.024063 13289 connection.cc:535] server connection from 127.1.121.66:57137 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 02:56:48.024430 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 13750
Aug 07 02:56:48 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:56:48.042652 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:48.025557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.1.121.67:37929)
0807 02:56:48.026161 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:48.026168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:48.026202 (+ 34us) server_negotiation.cc:407] Connection header received
0807 02:56:48.026266 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:48.026273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:48.026490 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:48.026635 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:48.028058 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:48.029352 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:48.030820 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:48.031479 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:48.031597 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:48.036271 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:48.036285 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:48.036305 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:48.036351 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:48.039834 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:48.040565 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:48.040573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:48.040575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:48.040722 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:48.041204 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:48.041208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:48.041210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:48.041609 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:48.041853 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:48.042115 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 02:56:48.042327 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":260,"threads_started":1}
I20260807 02:56:48.044292 13902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:48.130419 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 13750
I20260807 02:56:48.156389 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 13265
I20260807 02:56:48.157835 13353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:56:48.265849 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 13265
I20260807 02:56:48.294546 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45727
--webserver_interface=127.1.121.126
--webserver_port=45221
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:45727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:56:48.654768 13932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:48.655107 13932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:48.655208 13932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:48.662773 13932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:56:48.662901 13932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:48.662963 13932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:48.663033 13932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:56:48.663086 13932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:56:48.673542 13932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:45727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=45221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:48.675017 13932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:48.676431 13932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:48.689306 13939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:48.689357 13943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:48.689466 13938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:48.690613 13932 server_base.cc:1034] running on GCE node
I20260807 02:56:48.691426 13932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:48.693022 13932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:48.694253 13932 hybrid_clock.cc:648] HybridClock initialized: now 1786071408694214 us; error 31 us; skew 500 ppm
I20260807 02:56:48.694670 13932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:48 dist-test-slave-3h5h krb5kdc[13251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:48.697994 13932 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:56:48.699573 13932 webserver.cc:460] Webserver started at http://127.1.121.126:45221/ using document root <none> and password file <none>
I20260807 02:56:48.700299 13932 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:48.700464 13932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:48.706180 13932 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:56:48.709263 13951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:48.710752 13932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:56:48.710978 13932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef"
format_stamp: "Formatted at 2026-08-07 02:56:45 on dist-test-slave-3h5h"
server_key: "06b42ef8220c1c3bd539bc42c4c2552c"
server_key_iv: "4e03e0749285de4e48f6684b94096ecd"
server_key_version: "encryptionkey@0"
I20260807 02:56:48.712071 13932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:48.743394 13932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:48.747958 13932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:48.748281 13932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:48.769254 13932 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:45727
I20260807 02:56:48.769279 14013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:45727 every 8 connection(s)
I20260807 02:56:48.771096 13932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:56:48.773375 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 13932
I20260807 02:56:48.774334 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:38017
--local_ip_for_outbound_sockets=127.1.121.65
--tserver_master_addrs=127.1.121.126:45727
--webserver_port=32875
--webserver_interface=127.1.121.65
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.121.65
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:56:48.778548 14014 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:56:48.783687 14014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: Bootstrap starting.
I20260807 02:56:48.810707 14014 log.cc:824] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: Log is configured to *not* fsync() on all Append() calls
I20260807 02:56:48.822181 14014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:56:48.822837 14014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: Bootstrap complete.
I20260807 02:56:48.836202 14014 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } }
I20260807 02:56:48.837442 14014 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc06c1c601f14d9c8f8d5edf31b4b6ef, State: Initialized, Role: FOLLOWER
I20260807 02:56:48.838219 14014 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } }
I20260807 02:56:48.838526 14014 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:56:48.838652 14014 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:56:48.838855 14014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:56:48.844853 14014 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } }
I20260807 02:56:48.845521 14014 leader_election.cc:302] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc06c1c601f14d9c8f8d5edf31b4b6ef; no voters:
I20260807 02:56:48.846201 14014 leader_election.cc:288] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 02:56:48.846357 14018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 2 FOLLOWER]: Leader election won for term 2
I20260807 02:56:48.847597 14018 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [term 2 LEADER]: Becoming Leader. State: Replica: cc06c1c601f14d9c8f8d5edf31b4b6ef, State: Running, Role: LEADER
I20260807 02:56:48.848237 14018 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } }
I20260807 02:56:48.848389 14014 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:56:48.851079 14021 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc06c1c601f14d9c8f8d5edf31b4b6ef. Latest consensus state: current_term: 2 leader_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } } }
I20260807 02:56:48.851203 14020 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc06c1c601f14d9c8f8d5edf31b4b6ef" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45727 } } }
I20260807 02:56:48.851524 14021 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:48.851524 14020 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef [sys.catalog]: This master's current role is: LEADER
I20260807 02:56:48.853753 14025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:56:48.859404 14025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:56:48.860900 14025 catalog_manager.cc:1259] Loaded cluster ID: d3f7b72a2b6f4e669d7541f48f7f5d1e
I20260807 02:56:48.861066 14025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:56:48.865377 14025 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:56:48.866894 14025 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cc06c1c601f14d9c8f8d5edf31b4b6ef: Loaded TSK: 0
I20260807 02:56:48.868515 14025 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 02:56:49.160192 14016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:49.160495 14016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:49.160652 14016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:49.168107 14016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:49.168231 14016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:49.168430 14016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:56:49.178885 14016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.65.keytab
--principal=oryx/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:38017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=32875
--tserver_master_addrs=127.1.121.126:45727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:49.180330 14016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:49.181721 14016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:49.197043 14048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:49.197022 14047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:49.197865 14050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:49.197996 14016 server_base.cc:1034] running on GCE node
I20260807 02:56:49.198750 14016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:49.199767 14016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:49.201009 14016 hybrid_clock.cc:648] HybridClock initialized: now 1786071409200954 us; error 49 us; skew 500 ppm
I20260807 02:56:49.201365 14016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:49 dist-test-slave-3h5h krb5kdc[13251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071409, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:49.204711 14016 init.cc:377] Logged in from keytab as oryx/127.1.121.65@KRBTEST.COM (short username oryx)
I20260807 02:56:49.206401 14016 webserver.cc:460] Webserver started at http://127.1.121.65:32875/ using document root <none> and password file <none>
I20260807 02:56:49.207130 14016 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:49.207283 14016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:49.213284 14016 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:56:49.216248 14061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:49.217767 14016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:56:49.217952 14016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "91ba6ee7f7b54cc991f0955cc1eb7a34"
format_stamp: "Formatted at 2026-08-07 02:56:46 on dist-test-slave-3h5h"
server_key: "66871f675ef88c147866358cf73129dc"
server_key_iv: "dba44d2a329d7dd60867407153e7a86d"
server_key_version: "encryptionkey@0"
I20260807 02:56:49.218957 14016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:49.231191 14016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:49.235280 14016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:49.235574 14016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:49.236550 14016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:49.237931 14016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:49.238049 14016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:49.238183 14016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:49.238283 14016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:49.272112 14016 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:38017
I20260807 02:56:49.272171 14207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:38017 every 8 connection(s)
I20260807 02:56:49.273747 14016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:56:49.281384 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 14016
I20260807 02:56:49.282101 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:42285
--local_ip_for_outbound_sockets=127.1.121.66
--tserver_master_addrs=127.1.121.126:45727
--webserver_port=37591
--webserver_interface=127.1.121.66
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.121.66
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:56:49 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:49 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:56:49.641022 14213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:49.641330 14213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:49.641479 14213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:49.649014 14213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:49.649156 14213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:49.649346 14213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:56:49.659478 14213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.66.keytab
--principal=oryx/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:42285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=37591
--tserver_master_addrs=127.1.121.126:45727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:49.660946 14213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:49.662319 14213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:49.681337 14221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:49.681380 14220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:49.682132 14224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:49.687634 14213 server_base.cc:1034] running on GCE node
I20260807 02:56:49.688354 14213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:49.689569 14213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:49.691004 14213 hybrid_clock.cc:648] HybridClock initialized: now 1786071409690927 us; error 70 us; skew 500 ppm
I20260807 02:56:49.691354 14213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:49 dist-test-slave-3h5h krb5kdc[13251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071409, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:49.694761 14213 init.cc:377] Logged in from keytab as oryx/127.1.121.66@KRBTEST.COM (short username oryx)
I20260807 02:56:49.696557 14213 webserver.cc:460] Webserver started at http://127.1.121.66:37591/ using document root <none> and password file <none>
I20260807 02:56:49.697393 14213 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:49.697546 14213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:49.703943 14213 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:56:49.707393 14237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:49.709199 14213 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 02:56:49.709376 14213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "b1e04d475710407a9c3c7b7642542d73"
format_stamp: "Formatted at 2026-08-07 02:56:46 on dist-test-slave-3h5h"
server_key: "8e2d23817fa89c9dcce8f1b3c124d338"
server_key_iv: "4100fc3c60db97800aa4ca08605f11a2"
server_key_version: "encryptionkey@0"
I20260807 02:56:49.710374 14213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:49.733954 14213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:49.738593 14213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:49.738900 14213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:49.740042 14213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:49.741756 14213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:49.741937 14213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:49.742092 14213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:49.742187 14213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:49.803304 14213 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:42285
I20260807 02:56:49.803361 14382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:42285 every 8 connection(s)
I20260807 02:56:49.804960 14213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:56:49.809230 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 14213
I20260807 02:56:49.809962 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:37929
--local_ip_for_outbound_sockets=127.1.121.67
--tserver_master_addrs=127.1.121.126:45727
--webserver_port=37601
--webserver_interface=127.1.121.67
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.121.67
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:56:49 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:49 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:56:50.551249 14388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:56:50.551690 14388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:56:50.551944 14388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:56:50.561578 14388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:56:50.561693 14388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:56:50.561887 14388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:56:50.574512 14388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/krb5kdc/oryx_127.1.121.67.keytab
--principal=oryx/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:37929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=37601
--tserver_master_addrs=127.1.121.126:45727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:56:50.576148 14388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:56:50.577764 14388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:56:50.599208 14396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:50.600205 14395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:56:50.600406 14400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:56:50.602717 14388 server_base.cc:1034] running on GCE node
I20260807 02:56:50.603457 14388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:56:50.604735 14388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:56:50.608114 14388 hybrid_clock.cc:648] HybridClock initialized: now 1786071410608007 us; error 109 us; skew 500 ppm
I20260807 02:56:50.608505 14388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:56:50 dist-test-slave-3h5h krb5kdc[13251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071410, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:56:50.612304 14388 init.cc:377] Logged in from keytab as oryx/127.1.121.67@KRBTEST.COM (short username oryx)
I20260807 02:56:50.614375 14388 webserver.cc:460] Webserver started at http://127.1.121.67:37601/ using document root <none> and password file <none>
I20260807 02:56:50.615233 14388 fs_manager.cc:359] Metadata directory not provided
I20260807 02:56:50.615481 14388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:56:50.622601 14388 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260807 02:56:50.626556 14410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:50.628652 14388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 02:56:50.628919 14388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "1a5d8e52b6ca4577899b7980edf6dacd"
format_stamp: "Formatted at 2026-08-07 02:56:47 on dist-test-slave-3h5h"
server_key: "06889c38bc571f9928b28e9de3ab6458"
server_key_iv: "5b55052f6d6c18f1449741c6ec46ee03"
server_key_version: "encryptionkey@0"
I20260807 02:56:50.630231 14388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:56:50.650895 14388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:56:50.656153 14388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:56:50.656481 14388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:56:50.657860 14388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:56:50.659544 14388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:56:50.659720 14388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:50.659871 14388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:56:50.659978 14388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:56:50.721938 14388 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:37929
I20260807 02:56:50.721967 14548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:37929 every 8 connection(s)
I20260807 02:56:50.725530 14388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:56:50.731112 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 14388
Aug 07 02:56:50 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:50 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:50.777904 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:50.739372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.1.121.126:45727)
0807 02:56:50.746251 (+ 6879us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:50.746261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:50.746285 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:56:50.746353 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:50.746361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:50.746444 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:50.748590 (+ 2146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:50.750855 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.752070 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.755471 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.756421 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.758007 (+ 1586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:50.764100 (+ 6093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:56:50.764117 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:56:50.764153 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:56:50.764228 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:56:50.770297 (+ 6069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:50.773624 (+ 3327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:50.773636 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:50.773648 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:50.773924 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:56:50.774508 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:56:50.774515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:56:50.774520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:56:50.775231 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:56:50.775622 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0807 02:56:50.776143 (+ 521us) server_negotiation.cc:299] Negotiation successful
0807 02:56:50.776737 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6685,"thread_start_us":6520,"threads_started":1}
W20260807 02:56:50.816951 14212 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.527s user 0.013s sys 0.007s
W20260807 02:56:50.819350 14210 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:49.277242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:50185 (local address 127.1.121.126:45727)
0807 02:56:49.278031 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:49.278042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:49.278744 (+ 702us) server_negotiation.cc:407] Connection header received
0807 02:56:49.280962 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:49.280975 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:49.281299 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:49.281640 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:49.283552 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:49.285660 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:49.287365 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:49.288499 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:49.288700 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:50.818823 (+1530123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:50185: BlockingRecv error: recv got EOF from 127.1.121.65:50185 (error 108)
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":333,"threads_started":1}
I20260807 02:56:50.822183 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:49.277297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:50185)
0807 02:56:49.278561 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:49.278599 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:49.280345 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:49.282212 (+ 1867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:49.282225 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:49.282813 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:49.283334 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:49.283351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:49.285860 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:49.285865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:49.287205 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:49.287218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:49.288736 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:49.288741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:49.289016 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:49.289743 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:49.289767 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:50.820367 (+1530600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":830,"threads_started":1}
W20260807 02:56:50.823868 14208 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:45727 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.121.126:45727: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Aug 07 02:56:50 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:50 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:50.843449 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:50.824766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:43541)
0807 02:56:50.825543 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:50.825572 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:50.825753 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:50.826330 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:50.826338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:50.827034 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:50.827352 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:50.827370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.829781 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.829791 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:50.832116 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:50.832140 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.834087 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.834099 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:50.834427 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:50.835633 (+ 1206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:50.835675 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:50.842979 (+ 7304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549}
W20260807 02:56:50.843855 14210 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:50.825210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:43541 (local address 127.1.121.126:45727)
0807 02:56:50.825505 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:50.825516 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:50.825676 (+ 160us) server_negotiation.cc:407] Connection header received
0807 02:56:50.825960 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:50.825969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:50.826110 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:50.826267 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:50.827588 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.829553 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.832348 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.833496 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.833704 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:50.843398 (+ 9694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:43541: BlockingRecv error: recv got EOF from 127.1.121.65:43541 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 07 02:56:50 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:50 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:56:50.868335 14210 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:50.845325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:54107 (local address 127.1.121.126:45727)
0807 02:56:50.845630 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:50.845641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:50.849359 (+ 3718us) server_negotiation.cc:407] Connection header received
0807 02:56:50.849824 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:50.849834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:50.849952 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:50.850053 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:50.851834 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.853852 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.857848 (+ 3996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.858871 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.859059 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:50.867922 (+ 8863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:54107: BlockingRecv error: recv got EOF from 127.1.121.65:54107 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 02:56:50.869571 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:50.844726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:54107)
0807 02:56:50.849199 (+ 4473us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:50.849230 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:50.849588 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:50.850389 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:50.850398 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:50.851178 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:50.851557 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:50.851576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.854204 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.854214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:50.856983 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:50.857006 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.859464 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.859474 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:50.859813 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:50.860858 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:50.860897 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:50.869023 (+ 8126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4292,"thread_start_us":12600,"threads_started":1}
W20260807 02:56:50.870167 14208 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 02:56:51.024641 14387 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.200s user 0.011s sys 0.004s
W20260807 02:56:51.026225 14386 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:49.807879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:42987 (local address 127.1.121.126:45727)
0807 02:56:49.808449 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:49.808456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:49.813210 (+ 4754us) server_negotiation.cc:407] Connection header received
0807 02:56:49.813883 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:49.813892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:49.814014 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:49.814189 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:49.817616 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:49.818852 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:49.822484 (+ 3632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:49.823189 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:49.823344 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.025782 (+1202438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:42987: BlockingRecv error: recv got EOF from 127.1.121.66:42987 (error 108)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":269,"threads_started":1}
I20260807 02:56:51.026855 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:49.808367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:42987)
0807 02:56:49.809529 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:49.809614 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:49.813373 (+ 3759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:49.814629 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:49.814647 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:49.815986 (+ 1339us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:49.817310 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:49.817348 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:49.819073 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:49.819082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:49.822234 (+ 3152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:49.822258 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:49.823462 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:49.823472 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:49.823784 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:49.824572 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:49.824605 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:51.025532 (+1200927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":415,"threads_started":1}
W20260807 02:56:51.028144 14383 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:45727 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.121.126:45727: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:51.040964 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:51.029069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:59987)
0807 02:56:51.029324 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:51.029344 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:51.029474 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:51.029835 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:51.029840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:51.030349 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:51.030520 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.030529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.032752 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.032757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.033817 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.033830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.035310 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.035315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.035439 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.036168 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:51.036186 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:51.040676 (+ 4490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260807 02:56:51.042465 14386 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:51.029205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:59987 (local address 127.1.121.126:45727)
0807 02:56:51.029375 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:51.029383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:51.029403 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:56:51.029619 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:51.029624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:51.029707 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:51.029807 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:51.030759 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.032553 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.033981 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.035096 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.035281 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.041413 (+ 6132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:59987: BlockingRecv error: recv got EOF from 127.1.121.66:59987 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:51.055667 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:51.041970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:41545)
0807 02:56:51.042873 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:51.042901 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:51.043068 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:51.043518 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:51.043526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:51.044133 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:51.044434 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.044451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.045615 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.045624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.047914 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.047937 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.048746 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.048756 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.049115 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.049910 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:51.049946 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:51.055429 (+ 5483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":735}
W20260807 02:56:51.056107 14383 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 02:56:51.056185 14210 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:51.042160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:41545 (local address 127.1.121.126:45727)
0807 02:56:51.042363 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:51.042371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:51.042998 (+ 627us) server_negotiation.cc:407] Connection header received
0807 02:56:51.043226 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:51.043231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:51.043315 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:51.043424 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:51.044664 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.045441 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.048106 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.048534 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.048659 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.055807 (+ 7148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:41545: BlockingRecv error: recv got EOF from 127.1.121.66:41545 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:51.888505 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:51.870963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:47231)
0807 02:56:51.873267 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:51.873284 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:51.873450 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:51.873844 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:51.873848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:51.874316 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:51.874475 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.874485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.875696 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.875702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.876858 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.876872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.877604 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.877611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.877806 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.884257 (+ 6451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:51.884286 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:51.888162 (+ 3876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2184,"thread_start_us":2052,"threads_started":1}
W20260807 02:56:51.888685 14564 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:51.871091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:47231 (local address 127.1.121.126:45727)
0807 02:56:51.873135 (+ 2044us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:51.873143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:51.873380 (+ 237us) server_negotiation.cc:407] Connection header received
0807 02:56:51.873591 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:51.873598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:51.873677 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:51.873807 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:51.874639 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.875543 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.877000 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.877415 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.877554 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.888408 (+ 10854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:47231: BlockingRecv error: recv got EOF from 127.1.121.65:47231 (error 108)
Metrics: {"server-negotiator.queue_time_us":1898,"thread_start_us":1717,"threads_started":1}
W20260807 02:56:51.903520 14553 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.147s user 0.004s sys 0.021s
W20260807 02:56:51.904796 14555 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:50.732017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:56307 (local address 127.1.121.126:45727)
0807 02:56:50.739451 (+ 7434us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:50.739476 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:50.744883 (+ 5407us) server_negotiation.cc:407] Connection header received
0807 02:56:50.744996 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:50.745004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:50.745134 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:50.745295 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:50.748091 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.749553 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.753825 (+ 4272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.754522 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.754621 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.904571 (+1149950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:56307: BlockingRecv error: recv got EOF from 127.1.121.67:56307 (error 108)
Metrics: {"server-negotiator.queue_time_us":7249,"thread_start_us":7090,"threads_started":1}
I20260807 02:56:51.905354 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:50.730025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:56307)
0807 02:56:50.740311 (+ 10286us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:50.740386 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:50.744238 (+ 3852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:50.745770 (+ 1532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:50.745790 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:50.746944 (+ 1154us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:50.747825 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:50.747851 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.749914 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.749923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:50.753614 (+ 3691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:50.753632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:50.755053 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:50.755061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:50.755376 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:50.756303 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:50.756337 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:51.904317 (+1147980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9810,"thread_start_us":9524,"threads_started":1}
W20260807 02:56:51.906318 14549 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:45727 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.121.126:45727: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:51.916057 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:51.906960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:43585)
0807 02:56:51.907189 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:51.907212 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:51.907317 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:51.907692 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:51.907696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:51.908185 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:51.908369 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.908378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.909726 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.909731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.910674 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.910686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.911534 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.911538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.911654 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.912243 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:51.912260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:51.915849 (+ 3589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260807 02:56:51.916241 14555 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:51.907094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:43585 (local address 127.1.121.126:45727)
0807 02:56:51.907353 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:51.907360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:51.907389 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:56:51.907440 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:51.907443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:51.907541 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:51.907654 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:51.908592 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.909599 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.910843 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.911367 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.911486 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.916030 (+ 4544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:43585: BlockingRecv error: recv got EOF from 127.1.121.67:43585 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:51 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:51.925691 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:51.916831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:35173)
0807 02:56:51.916999 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:51.917020 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:51.917130 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:51.917477 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:51.917482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:51.917802 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:51.917955 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.917963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.918973 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.918978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.920145 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:51.920164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.921175 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.921183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:51.921411 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.921968 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:51.922091 (+ 123us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:51.925494 (+ 3403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 02:56:51.926000 14555 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:51.916928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:35173 (local address 127.1.121.126:45727)
0807 02:56:51.917113 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:51.917118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:51.917137 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:56:51.917252 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:51.917257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:51.917337 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:51.917446 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:51.918100 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.918824 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.920316 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:51.921023 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:51.921119 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:51.925674 (+ 4555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:35173: BlockingRecv error: recv got EOF from 127.1.121.67:35173 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260807 02:56:51.926051 14549 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 02:56:52 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:52 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:52.067360 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:52.056827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:36801)
0807 02:56:52.057653 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:52.057671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:52.057850 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:52.058279 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:52.058285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:52.058807 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:52.058974 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:52.058985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.060106 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.060113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:52.061217 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:52.061231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.062106 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.062114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:52.062299 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:52.062931 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:52.062957 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:52.066974 (+ 4017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":499,"threads_started":1}
W20260807 02:56:52.067454 14555 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:52.057017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:36801 (local address 127.1.121.126:45727)
0807 02:56:52.057286 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:52.057293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:52.057774 (+ 481us) server_negotiation.cc:407] Connection header received
0807 02:56:52.058045 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:52.058050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:52.058118 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:52.058239 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:52.059157 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.059923 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.061384 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.061884 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.062061 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:52.067218 (+ 5157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:36801: BlockingRecv error: recv got EOF from 127.1.121.66:36801 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Aug 07 02:56:52 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:52 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:52.899857 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:52.889833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:58201)
0807 02:56:52.890464 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:52.890486 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:52.890618 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:52.891155 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:52.891159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:52.891607 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:52.891802 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:52.891812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.893017 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.893021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:52.894171 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:52.894192 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.895129 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.895134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:52.895340 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:52.896011 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:52.896030 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:52.899604 (+ 3574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":375,"threads_started":1}
W20260807 02:56:52.900345 14569 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:52.889930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:58201 (local address 127.1.121.126:45727)
0807 02:56:52.890837 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:52.890841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:52.890858 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:56:52.890910 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:52.890913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:52.890982 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:52.891088 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:52.891962 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.892820 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.894348 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.894905 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.895063 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:52.899819 (+ 4756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:58201: BlockingRecv error: recv got EOF from 127.1.121.65:58201 (error 108)
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":647,"threads_started":1}
Aug 07 02:56:52 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:52 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:52.937353 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:52.926628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:37739)
0807 02:56:52.927234 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:52.927251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:52.927423 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:52.927807 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:52.927816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:52.928296 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:52.928448 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:52.928457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.929613 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.929619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:52.930828 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:52.930845 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.931939 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.931946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:52.932125 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:52.932737 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:52.932754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:52.937086 (+ 4332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":326,"threads_started":1}
W20260807 02:56:52.937438 14569 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:52.926734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:37739 (local address 127.1.121.126:45727)
0807 02:56:52.926993 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:52.927003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:52.927350 (+ 347us) server_negotiation.cc:407] Connection header received
0807 02:56:52.927560 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:52.927568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:52.927639 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:52.927750 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:52.928609 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.929454 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.931059 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:52.931718 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:52.931845 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:52.937261 (+ 5416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:37739: BlockingRecv error: recv got EOF from 127.1.121.67:37739 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 07 02:56:53 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:53 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:53.078591 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:53.068537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:45203)
0807 02:56:53.069132 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:53.069152 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:53.069284 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:53.069676 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:53.069680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:53.070275 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:53.070494 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:53.070504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.071653 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.071657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:53.072758 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:53.072772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.073535 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.073540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:53.073666 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:53.074262 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:53.074280 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:53.078227 (+ 3947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":350,"threads_started":1}
W20260807 02:56:53.078783 14569 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:53.068711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:45203 (local address 127.1.121.126:45727)
0807 02:56:53.068946 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:53.068950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:53.069242 (+ 292us) server_negotiation.cc:407] Connection header received
0807 02:56:53.069384 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:53.069388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:53.069451 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:53.069534 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:53.070656 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.071491 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.072950 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.073374 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.073471 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:53.078470 (+ 4999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:45203: BlockingRecv error: recv got EOF from 127.1.121.66:45203 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 07 02:56:53 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:53 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:53.911154 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:53.901162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:39097)
0807 02:56:53.901779 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:53.901800 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:53.901966 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:53.902478 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:53.902481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:53.903053 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:53.903270 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:53.903281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.904349 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.904354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:53.905534 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:53.905551 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.906269 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.906274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:53.906412 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:53.907062 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:53.907083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:53.910798 (+ 3715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":315,"threads_started":1}
W20260807 02:56:53.911252 14573 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:53.901274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:39097 (local address 127.1.121.126:45727)
0807 02:56:53.901807 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:53.901811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:53.901875 (+ 64us) server_negotiation.cc:407] Connection header received
0807 02:56:53.902110 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:53.902117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:53.902191 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:53.902283 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:53.903418 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.904176 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.905681 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.906125 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.906223 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:53.911014 (+ 4791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:39097: BlockingRecv error: recv got EOF from 127.1.121.65:39097 (error 108)
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":291,"threads_started":1}
Aug 07 02:56:53 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:53 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:53.949354 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:53.938664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:59139)
0807 02:56:53.939479 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:53.939516 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:53.939667 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:53.940103 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:53.940109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:53.940696 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:53.940973 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:53.940989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.942103 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.942108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:53.943323 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:53.943338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.944151 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.944157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:53.944316 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:53.945015 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:53.945035 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:53.949062 (+ 4027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":541,"threads_started":1}
W20260807 02:56:53.949697 14573 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:53.938776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:59139 (local address 127.1.121.126:45727)
0807 02:56:53.939081 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:53.939092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:53.939632 (+ 540us) server_negotiation.cc:407] Connection header received
0807 02:56:53.939793 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:53.939799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:53.939888 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:53.939991 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:53.941141 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.941939 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.943494 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:53.943938 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:53.944056 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:53.949376 (+ 5320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:59139: BlockingRecv error: recv got EOF from 127.1.121.67:59139 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 07 02:56:54 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:54 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:54.089759 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:54.079665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:34579)
0807 02:56:54.080372 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:54.080389 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:54.080507 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:54.080826 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:54.080851 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:54.081254 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:54.081424 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:54.081433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.082702 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.082706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:54.083588 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:54.083599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.084439 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.084443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:54.084562 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:54.085237 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:54.085256 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:54.089400 (+ 4144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":404,"threads_started":1}
W20260807 02:56:54.090031 14573 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:54.079854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:34579 (local address 127.1.121.126:45727)
0807 02:56:54.080143 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:54.080148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:54.080496 (+ 348us) server_negotiation.cc:407] Connection header received
0807 02:56:54.080614 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:54.080619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:54.080686 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:54.080784 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:54.081612 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.082535 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.083745 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.084301 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.084419 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:54.089727 (+ 5308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:34579: BlockingRecv error: recv got EOF from 127.1.121.66:34579 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 02:56:54 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:54 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:54.921782 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:54.912211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:57505)
0807 02:56:54.912716 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:54.912734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:54.912900 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:54.913429 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:54.913433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:54.913867 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:54.914061 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:54.914126 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.915118 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.915122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:54.916235 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:54.916247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.916984 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.916989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:54.917144 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:54.917714 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:54.917734 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:54.921419 (+ 3685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":263,"threads_started":1}
W20260807 02:56:54.922238 14578 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:54.912284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:57505 (local address 127.1.121.126:45727)
0807 02:56:54.913126 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:54.913130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:54.913146 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:56:54.913192 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:54.913195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:54.913251 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:54.913339 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:54.914267 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.914992 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.916392 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.916798 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.916948 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:54.921761 (+ 4813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:57505: BlockingRecv error: recv got EOF from 127.1.121.65:57505 (error 108)
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":612,"threads_started":1}
Aug 07 02:56:54 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:54 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:54.959815 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:54.950292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:46865)
0807 02:56:54.950897 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:54.950915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:54.951077 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:54.951385 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:54.951389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:54.951799 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:54.951983 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:54.951994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.953027 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.953033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:54.953969 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:54.953983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.954673 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.954678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:54.954835 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:54.955452 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:54.955471 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:54.959467 (+ 3996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":304,"threads_started":1}
W20260807 02:56:54.959944 14578 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:54.950358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:46865 (local address 127.1.121.126:45727)
0807 02:56:54.950669 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:54.950676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:54.950998 (+ 322us) server_negotiation.cc:407] Connection header received
0807 02:56:54.951191 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:54.951196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:54.951262 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:54.951346 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:54.952117 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.952881 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.954110 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:54.954499 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:54.954610 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:54.959705 (+ 5095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:46865: BlockingRecv error: recv got EOF from 127.1.121.67:46865 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Aug 07 02:56:55 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:55 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:55.103602 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:55.090883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:58621)
0807 02:56:55.091490 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:55.091507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:55.091680 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:55.092092 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:55.092099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:55.092665 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:55.093024 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:55.093042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.094623 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.094633 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:55.096567 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:55.096588 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.097888 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.097895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:55.098127 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:55.098895 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:55.098916 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:55.103155 (+ 4239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":370,"threads_started":1}
W20260807 02:56:55.103634 14578 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:55.090977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:58621 (local address 127.1.121.126:45727)
0807 02:56:55.091201 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:55.091210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:55.091613 (+ 403us) server_negotiation.cc:407] Connection header received
0807 02:56:55.091839 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:55.091844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:55.091926 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:55.092035 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:55.093259 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.094410 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.096778 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.097694 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.097834 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:55.103362 (+ 5528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:58621: BlockingRecv error: recv got EOF from 127.1.121.66:58621 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 02:56:55 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:55 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:55.932693 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:55.922789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:56917)
0807 02:56:55.923711 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:55.923734 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:55.923882 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:55.924475 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:55.924480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:55.925092 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:55.925349 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:55.925367 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.926471 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.926475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:55.927411 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:55.927422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.928169 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.928173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:55.928292 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:55.928904 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:55.928921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:55.932445 (+ 3524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":379,"threads_started":1}
W20260807 02:56:55.933102 14585 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:55.922920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:56917 (local address 127.1.121.126:45727)
0807 02:56:55.923796 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:55.923800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:55.923842 (+ 42us) server_negotiation.cc:407] Connection header received
0807 02:56:55.924170 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:55.924175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:55.924289 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:55.924388 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:55.925520 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.926334 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.927554 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.927957 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.928131 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:55.932704 (+ 4573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:56917: BlockingRecv error: recv got EOF from 127.1.121.65:56917 (error 108)
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":626,"threads_started":1}
Aug 07 02:56:55 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:55 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:55.971014 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:55.961401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:53367)
0807 02:56:55.962069 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:55.962088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:55.962252 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:55.962662 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:55.962666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:55.963085 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:55.963262 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:55.963271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.964263 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.964267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:55.965427 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:55.965441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.966126 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.966130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:55.966266 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:55.966863 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:55.966882 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:55.970744 (+ 3862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":383,"threads_started":1}
W20260807 02:56:55.971283 14585 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:55.961414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:53367 (local address 127.1.121.126:45727)
0807 02:56:55.961605 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:55.961612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:55.962197 (+ 585us) server_negotiation.cc:407] Connection header received
0807 02:56:55.962403 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:55.962411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:55.962495 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:55.962624 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:55.963386 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.964137 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.965583 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:55.965976 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:55.966073 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:55.970997 (+ 4924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:53367: BlockingRecv error: recv got EOF from 127.1.121.67:53367 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 02:56:56 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:56 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:56.115199 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:56.104790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:57637)
0807 02:56:56.105409 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:56.105428 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:56.105554 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:56.105907 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:56.105910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:56.106352 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:56.106546 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:56.106555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.107626 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.107629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:56.108717 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:56.108730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.109691 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.109698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:56.109923 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:56.110626 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:56.110673 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:56.114826 (+ 4153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":293,"threads_started":1}
W20260807 02:56:56.115532 14585 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:56.105134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:57637 (local address 127.1.121.126:45727)
0807 02:56:56.105370 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:56.105375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:56.105533 (+ 158us) server_negotiation.cc:407] Connection header received
0807 02:56:56.105688 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:56.105691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:56.105771 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:56.105857 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:56.106704 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.107482 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.108907 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.109491 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.109632 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:56.115214 (+ 5582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:57637: BlockingRecv error: recv got EOF from 127.1.121.66:57637 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 07 02:56:56 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:56 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:56.948181 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:56.933836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:42701)
0807 02:56:56.934360 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:56.934379 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:56.934604 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:56.935162 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:56.935168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:56.935598 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:56.935769 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:56.935779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.937001 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.937011 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:56.937942 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:56.937955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.938738 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.938745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:56.938950 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:56.943673 (+ 4723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:56.943695 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:56.947822 (+ 4127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":270,"threads_started":1}
W20260807 02:56:56.948601 14589 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:56.933972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:42701 (local address 127.1.121.126:45727)
0807 02:56:56.934749 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:56.934753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:56.934770 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:56:56.934825 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:56.934828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:56.934962 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:56.935112 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:56.935930 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.936865 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.938105 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.938556 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.938704 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:56.948097 (+ 9393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:42701: BlockingRecv error: recv got EOF from 127.1.121.65:42701 (error 108)
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":531,"threads_started":1}
Aug 07 02:56:56 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:56 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:56.981958 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:56.971974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:45975)
0807 02:56:56.972724 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:56.972746 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:56.972938 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:56.973337 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:56.973343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:56.974006 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:56.974179 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:56.974189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.975244 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.975251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:56.976218 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:56.976230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.977034 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.977040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:56.977222 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:56.977871 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:56.977889 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:56.981618 (+ 3729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":428,"threads_started":1}
W20260807 02:56:56.982053 14589 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:56.972174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:45975 (local address 127.1.121.126:45727)
0807 02:56:56.972414 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:56.972421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:56.972857 (+ 436us) server_negotiation.cc:407] Connection header received
0807 02:56:56.973091 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:56.973098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:56.973183 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:56.973294 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:56.974328 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.975109 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.976366 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:56.976876 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:56.977016 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:56.981832 (+ 4816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:45975: BlockingRecv error: recv got EOF from 127.1.121.67:45975 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 02:56:57 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:57 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:57.125823 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:57.116152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:59345)
0807 02:56:57.116659 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:57.116678 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:57.116808 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:57.117289 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:57.117297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:57.117784 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:57.117972 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:57.117981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.119107 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.119112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:57.120262 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:57.120275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.121227 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.121231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:57.121357 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:57.122004 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:57.122023 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:57.125576 (+ 3553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
W20260807 02:56:57.126015 14589 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:57.116294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:59345 (local address 127.1.121.126:45727)
0807 02:56:57.116538 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:57.116544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:57.116762 (+ 218us) server_negotiation.cc:407] Connection header received
0807 02:56:57.116978 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:57.116982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:57.117064 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:57.117227 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:57.118163 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.118924 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.120413 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.121072 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.121168 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:57.125742 (+ 4574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:59345: BlockingRecv error: recv got EOF from 127.1.121.66:59345 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 07 02:56:57 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:57 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:57.959293 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:57.949334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:36651)
0807 02:56:57.950310 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:57.950327 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:57.950487 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:57.950954 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:57.950959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:57.951451 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:57.951725 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:57.951738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.952885 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.952889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:57.953868 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:57.953881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.954700 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.954705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:57.954823 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:57.955446 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:57.955463 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:57.959049 (+ 3586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":488,"threads_started":1}
W20260807 02:56:57.959895 14592 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:57.949436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:36651 (local address 127.1.121.126:45727)
0807 02:56:57.950226 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:57.950232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:57.950414 (+ 182us) server_negotiation.cc:407] Connection header received
0807 02:56:57.950674 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:57.950680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:57.950771 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:57.950905 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:57.951874 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.952714 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.954033 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.954556 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.954680 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:57.959378 (+ 4698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:36651: BlockingRecv error: recv got EOF from 127.1.121.65:36651 (error 108)
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":357,"threads_started":1}
Aug 07 02:56:57 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:57 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:57.994132 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:57.983035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:36495)
0807 02:56:57.983704 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:57.983730 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:57.983857 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:57.984269 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:57.984273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:57.984773 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:57.985021 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:57.985032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.986331 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.986337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:57.987867 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:57.987887 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.989043 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.989049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:57.989203 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:57.989886 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:57.989906 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:57.993780 (+ 3874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":333,"threads_started":1}
W20260807 02:56:57.994289 14592 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:57.983138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:36495 (local address 127.1.121.126:45727)
0807 02:56:57.983330 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:57.983335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:57.983807 (+ 472us) server_negotiation.cc:407] Connection header received
0807 02:56:57.984026 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:57.984031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:57.984103 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:57.984205 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:57.985216 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.986153 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.988159 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:57.988873 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:57.989004 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:57.993993 (+ 4989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:36495: BlockingRecv error: recv got EOF from 127.1.121.67:36495 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 02:56:58 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:58 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:58.136605 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:58.126844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:50271)
0807 02:56:58.127412 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:58.127429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:58.127563 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:58.127885 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:58.127888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:58.128371 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:58.128567 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:58.128577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.129629 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.129634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:58.130613 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:58.130625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.131285 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.131290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:58.131410 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:58.132081 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:58.132099 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:58.136295 (+ 4196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":267,"threads_started":1}
W20260807 02:56:58.136747 14592 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:58.126889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:50271 (local address 127.1.121.126:45727)
0807 02:56:58.127187 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:58.127197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:58.127514 (+ 317us) server_negotiation.cc:407] Connection header received
0807 02:56:58.127672 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:58.127676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:58.127753 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:58.127837 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:58.128716 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.129496 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.130742 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.131144 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.131237 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:58.136482 (+ 5245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:50271: BlockingRecv error: recv got EOF from 127.1.121.66:50271 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 07 02:56:58 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:58 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:58.971153 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:58.960471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:37737)
0807 02:56:58.961215 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:58.961238 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:58.961380 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:58.961755 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:58.961760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:58.962259 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:58.962440 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:58.962450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.963636 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.963641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:58.964957 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:58.964973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.965824 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.965830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:58.965975 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:58.966698 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:58.966718 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:58.970745 (+ 4027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":337,"threads_started":1}
W20260807 02:56:58.971230 14597 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:58.960644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:37737 (local address 127.1.121.126:45727)
0807 02:56:58.961374 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:58.961380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:58.961402 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:56:58.961494 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:58.961500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:58.961591 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:58.961714 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:58.962614 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.963429 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.965144 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.965623 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.965720 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:58.971031 (+ 5311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:37737: BlockingRecv error: recv got EOF from 127.1.121.65:37737 (error 108)
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":465,"threads_started":1}
Aug 07 02:56:58 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:58 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:56:59.006392 14597 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:58.995314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:48179 (local address 127.1.121.126:45727)
0807 02:56:58.995596 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:58.995603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:58.995874 (+ 271us) server_negotiation.cc:407] Connection header received
0807 02:56:58.996182 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:58.996188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:58.996263 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:58.996378 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:58.997377 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.998180 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.999918 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.000561 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.000705 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:59.006127 (+ 5422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:48179: BlockingRecv error: recv got EOF from 127.1.121.67:48179 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260807 02:56:59.006386 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:58.995050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:48179)
0807 02:56:58.995759 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:58.995777 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:58.996022 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:58.996416 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:58.996422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:58.997022 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:58.997215 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:58.997225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:58.998343 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:58.998350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:58.999746 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:58.999761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.000734 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.000741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:59.000987 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:59.001673 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:59.001692 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:59.005888 (+ 4196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":346,"threads_started":1}
Aug 07 02:56:59 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:59 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:59.148469 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:59.137761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:55043)
0807 02:56:59.138314 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:59.138331 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:59.138452 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:59.138801 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:59.138804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:59.139211 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:59.139375 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:59.139383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.140750 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.140754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:59.141764 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:59.141776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.142599 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.142604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:59.142726 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:59.143392 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:59.143413 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:59.147920 (+ 4507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":297,"threads_started":1}
W20260807 02:56:59.148619 14597 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:59.137883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:55043 (local address 127.1.121.126:45727)
0807 02:56:59.138151 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:59.138158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:59.138419 (+ 261us) server_negotiation.cc:407] Connection header received
0807 02:56:59.138563 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:59.138567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:59.138658 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:59.138756 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:59.139528 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.140606 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.141909 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.142403 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.142514 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:59.148306 (+ 5792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:55043: BlockingRecv error: recv got EOF from 127.1.121.66:55043 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 07 02:56:59 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:56:59 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:56:59.982511 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:56:59.972265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:60245)
0807 02:56:59.972928 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0807 02:56:59.972952 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:56:59.973133 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:56:59.973498 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:56:59.973502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:56:59.973958 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:56:59.974131 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:59.974140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.975232 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.975237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:59.976386 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:56:59.976400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.977126 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.977131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:56:59.977265 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:59.977862 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:56:59.977881 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:56:59.982265 (+ 4384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":347,"threads_started":1}
W20260807 02:56:59.982764 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:56:59.972357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:60245 (local address 127.1.121.126:45727)
0807 02:56:59.972950 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0807 02:56:59.972955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:56:59.973052 (+ 97us) server_negotiation.cc:407] Connection header received
0807 02:56:59.973268 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:56:59.973272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:56:59.973334 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:56:59.973427 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:56:59.974267 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.975080 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.976532 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:56:59.976975 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:56:59.977078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:56:59.982503 (+ 5425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:60245: BlockingRecv error: recv got EOF from 127.1.121.65:60245 (error 108)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":316,"threads_started":1}
Aug 07 02:57:00 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:00 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:00.018872 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:00.007421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:45491)
0807 02:57:00.008170 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:00.008188 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:00.008383 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:00.008790 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:00.008796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:00.009362 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:00.009698 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:00.009715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.010975 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.010980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:00.012074 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:00.012089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.013012 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.013020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:00.013173 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:00.013938 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:00.013957 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:00.018582 (+ 4625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":464,"threads_started":1}
W20260807 02:57:00.019120 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:00.007535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:45491 (local address 127.1.121.126:45727)
0807 02:57:00.007781 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:00.007791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:00.008297 (+ 506us) server_negotiation.cc:407] Connection header received
0807 02:57:00.008557 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:00.008562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:00.008632 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:00.008751 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:00.009904 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.010804 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.012240 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.012737 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.012939 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:00.018801 (+ 5862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:45491: BlockingRecv error: recv got EOF from 127.1.121.67:45491 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 02:57:00 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:00 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:00.160301 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:00.149735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:49157)
0807 02:57:00.150334 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:00.150361 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:00.150505 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:00.150918 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:00.150924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:00.151563 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:00.151785 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:00.151801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.153242 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.153248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:00.154295 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:00.154310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.155196 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.155201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:00.155350 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:00.155968 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:00.156051 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:00.160038 (+ 3987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":327,"threads_started":1}
W20260807 02:57:00.160693 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:00.149873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:49157 (local address 127.1.121.126:45727)
0807 02:57:00.150200 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:00.150205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:00.150476 (+ 271us) server_negotiation.cc:407] Connection header received
0807 02:57:00.150634 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:00.150638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:00.150726 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:00.150822 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:00.152071 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.153025 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.154453 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.154936 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.155050 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:00.160370 (+ 5320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:49157: BlockingRecv error: recv got EOF from 127.1.121.66:49157 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 02:57:00 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:00 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:00.993927 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:00.983587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:37753)
0807 02:57:00.984177 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:00.984194 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:00.984313 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:00.984816 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:00.984819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:00.985253 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:00.985413 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:00.985422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.986602 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.986607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:00.987753 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:00.987767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.988696 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.988701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:00.988815 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:00.989749 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:00.989767 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:00.993615 (+ 3848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":330,"threads_started":1}
W20260807 02:57:00.994045 14608 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:00.983811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:37753 (local address 127.1.121.126:45727)
0807 02:57:00.984488 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:00.984494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:00.984515 (+ 21us) server_negotiation.cc:407] Connection header received
0807 02:57:00.984578 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:00.984582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:00.984652 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:00.984773 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:00.985598 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.986422 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.987897 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:00.988526 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:00.988660 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:00.993852 (+ 5192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:37753: BlockingRecv error: recv got EOF from 127.1.121.65:37753 (error 108)
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":370,"threads_started":1}
Aug 07 02:57:01 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:01 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:01.031330 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:01.020202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:43813)
0807 02:57:01.020912 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:01.020943 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:01.021079 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:01.021424 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:01.021428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:01.021934 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:01.022134 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:01.022145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.023543 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.023550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:01.024807 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:01.024827 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.026186 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.026200 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:01.026384 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:01.027166 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:01.027184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:01.031091 (+ 3907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":370,"threads_started":1}
W20260807 02:57:01.031620 14608 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:01.020386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:43813 (local address 127.1.121.126:45727)
0807 02:57:01.020579 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:01.020604 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:01.021048 (+ 444us) server_negotiation.cc:407] Connection header received
0807 02:57:01.021192 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:01.021196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:01.021264 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:01.021361 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:01.022289 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.023290 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.025091 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.025810 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.026009 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:01.031337 (+ 5328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:43813: BlockingRecv error: recv got EOF from 127.1.121.67:43813 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 02:57:01 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:01 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:01.171864 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:01.161691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:38961)
0807 02:57:01.162464 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:01.162488 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:01.162684 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:01.163179 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:01.163184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:01.163796 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:01.164047 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:01.164058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.165136 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.165140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:01.166264 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:01.166277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.166951 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.166956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:01.167158 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:01.167803 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:01.167822 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:01.171523 (+ 3701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":464,"threads_started":1}
W20260807 02:57:01.172030 14608 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:01.161819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:38961 (local address 127.1.121.126:45727)
0807 02:57:01.162071 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:01.162078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:01.162629 (+ 551us) server_negotiation.cc:407] Connection header received
0807 02:57:01.162859 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:01.162864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:01.162949 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:01.163056 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:01.164190 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.164962 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.166404 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.166816 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.166910 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:01.171749 (+ 4839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:38961: BlockingRecv error: recv got EOF from 127.1.121.66:38961 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 07 02:57:02 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:02 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:02.010990 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:01.995350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:50983)
0807 02:57:01.996003 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:01.996022 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:01.996137 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:01.996470 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:01.996474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:01.997175 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:01.997387 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:01.997397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.998449 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.998453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:01.999503 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:01.999517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.000362 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.000367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:02.000520 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:02.006516 (+ 5996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:02.006539 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:02.010622 (+ 4083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":386,"threads_started":1}
W20260807 02:57:02.011227 14615 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:01.995538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:50983 (local address 127.1.121.126:45727)
0807 02:57:01.996123 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:01.996128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:01.996145 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:57:01.996257 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:01.996260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:01.996319 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:01.996410 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:01.997537 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:01.998312 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:01.999636 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.000201 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.000298 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:02.010957 (+ 10659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:50983: BlockingRecv error: recv got EOF from 127.1.121.65:50983 (error 108)
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":306,"threads_started":1}
Aug 07 02:57:02 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:02 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:02.043337 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:02.032287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:57053)
0807 02:57:02.032908 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:02.032925 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:02.033161 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:02.033543 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:02.033550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:02.034017 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:02.034199 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:02.034210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.035431 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.035437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:02.036561 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:02.036574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.037465 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.037472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:02.037676 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:02.038392 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:02.038422 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:02.042992 (+ 4570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":346,"threads_started":1}
W20260807 02:57:02.043522 14615 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:02.032365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:57053 (local address 127.1.121.126:45727)
0807 02:57:02.032616 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:02.032625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:02.033075 (+ 450us) server_negotiation.cc:407] Connection header received
0807 02:57:02.033306 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:02.033317 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:02.033384 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:02.033502 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:02.034374 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.035261 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.036731 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.037277 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.037438 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:02.043273 (+ 5835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:57053: BlockingRecv error: recv got EOF from 127.1.121.67:57053 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 07 02:57:02 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:02 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:02.184132 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:02.172991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:51937)
0807 02:57:02.173705 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:02.173727 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:02.173873 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:02.174444 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:02.174451 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:02.175044 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:02.175265 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:02.175276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.176496 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.176501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:02.177626 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:02.177640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.178603 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.178610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:02.178820 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:02.179820 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:02.179847 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:02.183856 (+ 4009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":415,"threads_started":1}
W20260807 02:57:02.184558 14615 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:02.173069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:51937 (local address 127.1.121.126:45727)
0807 02:57:02.173254 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:02.173259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:02.173819 (+ 560us) server_negotiation.cc:407] Connection header received
0807 02:57:02.174157 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:02.174161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:02.174251 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:02.174364 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:02.175446 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.176327 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.177818 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:02.178398 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:02.178555 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:02.184190 (+ 5635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:51937: BlockingRecv error: recv got EOF from 127.1.121.66:51937 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 02:57:03 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:03 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:03.023687 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:03.012361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:53503)
0807 02:57:03.013291 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:03.013313 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:03.013507 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:03.014059 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:03.014064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:03.014540 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:03.014726 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:03.014735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.015947 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.015951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:03.017332 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:03.017346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.018191 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.018196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:03.018334 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:03.018990 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:03.019008 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:03.023254 (+ 4246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":648,"threads_started":1}
W20260807 02:57:03.024103 14622 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:03.012449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:53503 (local address 127.1.121.126:45727)
0807 02:57:03.013636 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:03.013643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:03.013668 (+ 25us) server_negotiation.cc:407] Connection header received
0807 02:57:03.013765 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:03.013770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:03.013853 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:03.014029 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:03.014861 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.015780 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.017551 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.018020 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.018142 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:03.023605 (+ 5463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:53503: BlockingRecv error: recv got EOF from 127.1.121.65:53503 (error 108)
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":894,"threads_started":1}
Aug 07 02:57:03 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:03 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:03.055269 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:03.044390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:59039)
0807 02:57:03.045025 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:03.045042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:03.045212 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:03.045555 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:03.045558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:03.046003 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:03.046193 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:03.046202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.047311 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.047316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:03.048320 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:03.048334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.049359 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.049364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:03.049619 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:03.050466 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:03.050489 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:03.055036 (+ 4547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":334,"threads_started":1}
W20260807 02:57:03.055755 14622 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:03.044511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:59039 (local address 127.1.121.126:45727)
0807 02:57:03.044784 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:03.044792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:03.045167 (+ 375us) server_negotiation.cc:407] Connection header received
0807 02:57:03.045350 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:03.045354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:03.045420 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:03.045508 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:03.046349 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.047157 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.048537 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.049186 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.049303 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:03.055279 (+ 5976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:59039: BlockingRecv error: recv got EOF from 127.1.121.67:59039 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Aug 07 02:57:03 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:03 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:03.194911 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:03.185281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:37739)
0807 02:57:03.185889 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:03.185905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:03.186128 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:03.186540 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:03.186546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:03.187023 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:03.187207 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:03.187218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.188286 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.188292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:03.189264 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:03.189276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.190068 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.190073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:03.190273 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:03.190954 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:03.190973 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:03.194549 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":340,"threads_started":1}
W20260807 02:57:03.195081 14622 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:03.185373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:37739 (local address 127.1.121.126:45727)
0807 02:57:03.185679 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:03.185688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:03.186008 (+ 320us) server_negotiation.cc:407] Connection header received
0807 02:57:03.186303 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:03.186308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:03.186390 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:03.186505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:03.187372 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.188131 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.189438 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:03.189871 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:03.190021 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:03.194809 (+ 4788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:37739: BlockingRecv error: recv got EOF from 127.1.121.66:37739 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Aug 07 02:57:04 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:04 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:04.034797 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:04.024761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:38357)
0807 02:57:04.025504 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:04.025523 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:04.025642 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:04.026033 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:04.026037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:04.026463 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:04.026630 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:04.026640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.027825 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.027829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:04.028735 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:04.028748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.029480 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.029485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:04.029633 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:04.030292 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:04.030309 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:04.034408 (+ 4099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":331,"threads_started":1}
W20260807 02:57:04.034934 14628 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:04.024894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:38357 (local address 127.1.121.126:45727)
0807 02:57:04.025567 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:04.025572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:04.025592 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:57:04.025769 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:04.025773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:04.025848 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:04.025949 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:04.026801 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.027667 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.028901 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.029340 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.029440 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:04.034669 (+ 5229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:38357: BlockingRecv error: recv got EOF from 127.1.121.65:38357 (error 108)
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":419,"threads_started":1}
Aug 07 02:57:04 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:04 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:04.067936 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:04.056564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:38549)
0807 02:57:04.057515 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:04.057537 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:04.057696 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:04.058180 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:04.058186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:04.058734 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:04.058929 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:04.058939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.060122 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.060128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:04.061755 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:04.061770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.062650 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.062657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:04.062864 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:04.063503 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:04.063522 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:04.067500 (+ 3978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":624,"threads_started":1}
W20260807 02:57:04.068007 14628 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:04.056591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:38549 (local address 127.1.121.126:45727)
0807 02:57:04.056855 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:04.056864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:04.057825 (+ 961us) server_negotiation.cc:407] Connection header received
0807 02:57:04.057889 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:04.057893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:04.057961 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:04.058136 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:04.059121 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.059923 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.061926 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.062412 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.062568 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:04.067752 (+ 5184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:38549: BlockingRecv error: recv got EOF from 127.1.121.67:38549 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 07 02:57:04 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:04 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:04.209056 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:04.196060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:40305)
0807 02:57:04.196610 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:04.196627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:04.196742 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:04.197314 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:04.197323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:04.197835 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:04.197987 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:04.197996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.200069 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.200074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:04.201292 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:04.201306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.202401 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.202406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:04.202549 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:04.203271 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:04.203292 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:04.208693 (+ 5401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":301,"threads_started":1}
W20260807 02:57:04.209278 14628 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:04.196279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:40305 (local address 127.1.121.126:45727)
0807 02:57:04.196519 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:04.196524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:04.196769 (+ 245us) server_negotiation.cc:407] Connection header received
0807 02:57:04.196912 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:04.196921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:04.197124 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:04.197223 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:04.198256 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.199788 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.201462 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:04.202219 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:04.202380 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:04.208966 (+ 6586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:40305: BlockingRecv error: recv got EOF from 127.1.121.66:40305 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 07 02:57:05 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:05 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.65@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:05.046257 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:05.036381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.65:34707)
0807 02:57:05.037044 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:05.037379 (+ 335us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:05.037520 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:05.038001 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:05.038006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:05.038542 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:05.038734 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:05.038744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.039882 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.039887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:05.040925 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:05.040939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.041670 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.041675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:05.041810 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:05.042420 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:05.042439 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:05.046014 (+ 3575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":308,"threads_started":1}
W20260807 02:57:05.046489 14632 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:05.036484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:34707 (local address 127.1.121.126:45727)
0807 02:57:05.037096 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:05.037102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:05.037484 (+ 382us) server_negotiation.cc:407] Connection header received
0807 02:57:05.037672 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:05.037678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:05.037788 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:05.037908 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:05.038892 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.039723 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.041065 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.041518 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.041648 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:05.046246 (+ 4598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.65:34707: BlockingRecv error: recv got EOF from 127.1.121.65:34707 (error 108)
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":331,"threads_started":1}
Aug 07 02:57:05 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:05 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.67@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:05.079530 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:05.069147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.67:35963)
0807 02:57:05.069747 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:05.069764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:05.069893 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:05.070334 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:05.070338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:05.070778 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:05.070936 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:05.070944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.072412 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.072417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:05.073487 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:05.073503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.074283 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.074288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:05.074467 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:05.075163 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:05.075182 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:05.079203 (+ 4021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":333,"threads_started":1}
W20260807 02:57:05.079632 14632 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:05.069232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:35963 (local address 127.1.121.126:45727)
0807 02:57:05.069538 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:05.069544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:05.069852 (+ 308us) server_negotiation.cc:407] Connection header received
0807 02:57:05.070037 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:05.070041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:05.070147 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:05.070234 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:05.071209 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.072257 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.073661 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.074114 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.074258 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:05.079426 (+ 5168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.67:35963: BlockingRecv error: recv got EOF from 127.1.121.67:35963 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
Aug 07 02:57:05 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:57:05 dist-test-slave-3h5h krb5kdc[13251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.121.66@KRBTEST.COM for oryx/127.1.121.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:57:05.220803 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:05.210702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45727 (local address 127.1.121.66:55195)
0807 02:57:05.211279 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:05.211298 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:05.211415 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:05.211795 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:05.211800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:05.212206 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:05.212454 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:05.212470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.213730 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.213736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:05.214796 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:05.214809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.215554 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.215560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:05.215765 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:05.216583 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:05.216604 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:05.220509 (+ 3905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.121.126:45727: Server oryx/127.1.121.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":315,"threads_started":1}
W20260807 02:57:05.221467 14632 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:05.210914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:55195 (local address 127.1.121.126:45727)
0807 02:57:05.211123 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:05.211129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:05.211394 (+ 265us) server_negotiation.cc:407] Connection header received
0807 02:57:05.211536 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:05.211540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:05.211617 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:05.211738 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:05.212617 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.213564 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.214959 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.215384 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.215527 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:05.220759 (+ 5232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.121.66:55195: BlockingRecv error: recv got EOF from 127.1.121.66:55195 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 02:57:05.743224 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 14016
I20260807 02:57:05.756829 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:05.744428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.1.121.65:38017)
0807 02:57:05.745164 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:05.745172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:05.745201 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:57:05.745273 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:05.745280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:05.745521 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:05.745694 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:05.747388 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.748459 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.749954 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.750623 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.750745 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:05.753661 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:05.753670 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:05.753703 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:05.753757 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:05.756279 (+ 2522us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.121.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 02:57:05.756603 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41266: Request ticket server kudu/127.1.121.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":397,"threads_started":1}
W20260807 02:57:05.756999 14636 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41266: Request ticket server kudu/127.1.121.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 02:57:05.757112 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 14016: Not authorized: Client connection negotiation failed: client connection to 127.1.121.65:38017: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.121.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 02:57:05.757406 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 14016
I20260807 02:57:05.792152 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 14213
I20260807 02:57:05.807346 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:05.793551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.1.121.66:42285)
0807 02:57:05.794276 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:05.794285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:05.794313 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:57:05.794382 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:05.794390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:05.794651 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:05.794899 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:05.796448 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.797741 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.799185 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.799828 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.800001 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:05.802996 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:05.803007 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:05.803023 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:05.803067 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:05.806588 (+ 3521us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.121.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 02:57:05.807017 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35780: Request ticket server kudu/127.1.121.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":387,"threads_started":1}
W20260807 02:57:05.807340 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 14213: Not authorized: Client connection negotiation failed: client connection to 127.1.121.66:42285: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.121.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 02:57:05.807482 14637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35780: Request ticket server kudu/127.1.121.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 02:57:05.807516 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 14213
I20260807 02:57:05.837304 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 14388
I20260807 02:57:05.853768 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:05.838881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.1.121.67:37929)
0807 02:57:05.839592 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:05.839600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:05.839629 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:57:05.839708 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:05.839716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:05.839984 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:05.840201 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:05.842874 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.844504 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.846395 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:05.846959 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:05.847083 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:05.850165 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:05.850175 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:05.850191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:05.850236 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:05.853168 (+ 2932us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.121.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 02:57:05.853484 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50380: Request ticket server kudu/127.1.121.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":331,"threads_started":1}
W20260807 02:57:05.853905 14638 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50380: Request ticket server kudu/127.1.121.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 02:57:05.853899 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 14388: Not authorized: Client connection negotiation failed: client connection to 127.1.121.67:37929: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.121.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 02:57:05.854091 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 14388
I20260807 02:57:05.883054 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 13932
I20260807 02:57:05.884634 14009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:06.007216 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 13932
2026-08-07T02:57:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22422 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T02:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:57:06Z Disabled control of system clock
I20260807 02:57:06.110839 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45561
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:45561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 02:57:06.484498 14643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:06.484813 14643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:06.484990 14643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:06.492331 14643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:57:06.492441 14643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:06.492480 14643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:06.492542 14643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:57:06.492587 14643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:57:06.502090 14643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:45561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:45561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:06.503407 14643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:06.505290 14643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:06.519236 14655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:06.519133 14649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:06.519197 14651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:06.519883 14643 server_base.cc:1034] running on GCE node
I20260807 02:57:06.520803 14643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:06.522580 14643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:06.523838 14643 hybrid_clock.cc:648] HybridClock initialized: now 1786071426523802 us; error 33 us; skew 500 ppm
I20260807 02:57:06.524232 14643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:57:06.526862 14643 webserver.cc:460] Webserver started at http://127.1.121.126:35099/ using document root <none> and password file <none>
I20260807 02:57:06.527612 14643 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:06.527755 14643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:06.528116 14643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:06.531175 14643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "a7ab1bc97d7449179fc97c566aeb6f81"
format_stamp: "Formatted at 2026-08-07 02:57:06 on dist-test-slave-3h5h"
server_key: "b1b8903ea68b34485478a05fc709d4d4"
server_key_iv: "7b6afac981ff3d34bc6a06910296ca4d"
server_key_version: "encryptionkey@0"
I20260807 02:57:06.532007 14643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "a7ab1bc97d7449179fc97c566aeb6f81"
format_stamp: "Formatted at 2026-08-07 02:57:06 on dist-test-slave-3h5h"
server_key: "b1b8903ea68b34485478a05fc709d4d4"
server_key_iv: "7b6afac981ff3d34bc6a06910296ca4d"
server_key_version: "encryptionkey@0"
I20260807 02:57:06.537482 14643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 02:57:06.541221 14663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:06.542619 14643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 02:57:06.542819 14643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "a7ab1bc97d7449179fc97c566aeb6f81"
format_stamp: "Formatted at 2026-08-07 02:57:06 on dist-test-slave-3h5h"
server_key: "b1b8903ea68b34485478a05fc709d4d4"
server_key_iv: "7b6afac981ff3d34bc6a06910296ca4d"
server_key_version: "encryptionkey@0"
I20260807 02:57:06.543041 14643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:06.569288 14643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:06.570200 14643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:06.570525 14643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:06.591423 14643 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:45561
I20260807 02:57:06.591521 14729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:45561 every 8 connection(s)
I20260807 02:57:06.592897 14643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:57:06.597422 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:57:06.601577 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 14643
I20260807 02:57:06.601807 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:57:06.602499 1509 external_mini_cluster.cc:1428] Setting key 9b92ba148ca11e627e528a75ed23fefe
I20260807 02:57:06.613528 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81: Bootstrap starting.
I20260807 02:57:06.618297 14731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81: Neither blocks nor log segments found. Creating new log.
I20260807 02:57:06.619884 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:06.604907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.1.121.126:45561)
0807 02:57:06.606130 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:06.606146 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:06.606204 (+ 58us) server_negotiation.cc:407] Connection header received
0807 02:57:06.608141 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:06.608176 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:06.608920 (+ 744us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:06.609415 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:06.610042 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:06.612278 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:06.613998 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:06.615149 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:06.615359 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:06.616571 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:06.616582 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:06.616601 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:06.617417 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:06.617566 (+ 149us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:06.617596 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:06.617726 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:06.617977 (+ 251us) server_negotiation.cc:299] Negotiation successful
0807 02:57:06.618368 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":372,"threads_started":1}
I20260807 02:57:06.619868 14731 log.cc:824] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81: Log is configured to *not* fsync() on all Append() calls
I20260807 02:57:06.624068 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81: No bootstrap required, opened a new log
I20260807 02:57:06.632318 14731 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ab1bc97d7449179fc97c566aeb6f81" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45561 } }
I20260807 02:57:06.632668 14731 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:57:06.632745 14731 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ab1bc97d7449179fc97c566aeb6f81, State: Initialized, Role: FOLLOWER
I20260807 02:57:06.633316 14731 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ab1bc97d7449179fc97c566aeb6f81" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45561 } }
I20260807 02:57:06.633582 14731 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:57:06.633709 14731 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:57:06.633841 14731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:57:06.636451 14731 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ab1bc97d7449179fc97c566aeb6f81" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45561 } }
I20260807 02:57:06.636991 14731 leader_election.cc:302] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7ab1bc97d7449179fc97c566aeb6f81; no voters:
I20260807 02:57:06.637542 14731 leader_election.cc:288] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:57:06.637758 14739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:57:06.638476 14739 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [term 1 LEADER]: Becoming Leader. State: Replica: a7ab1bc97d7449179fc97c566aeb6f81, State: Running, Role: LEADER
I20260807 02:57:06.639176 14739 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ab1bc97d7449179fc97c566aeb6f81" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45561 } }
I20260807 02:57:06.639995 14731 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:57:06.642295 14741 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7ab1bc97d7449179fc97c566aeb6f81. Latest consensus state: current_term: 1 leader_uuid: "a7ab1bc97d7449179fc97c566aeb6f81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ab1bc97d7449179fc97c566aeb6f81" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45561 } } }
I20260807 02:57:06.642383 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7ab1bc97d7449179fc97c566aeb6f81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ab1bc97d7449179fc97c566aeb6f81" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 45561 } } }
I20260807 02:57:06.642795 14741 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [sys.catalog]: This master's current role is: LEADER
I20260807 02:57:06.642829 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81 [sys.catalog]: This master's current role is: LEADER
I20260807 02:57:06.644521 14748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:57:06.650595 14748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:57:06.659734 14748 catalog_manager.cc:1347] Generated new cluster ID: b6cb09f3e5854612a6b6a7cfe7f7b208
I20260807 02:57:06.659899 14748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:57:06.674588 14748 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:57:06.676424 14748 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:57:06.694607 14748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7ab1bc97d7449179fc97c566aeb6f81: Generated new TSK 0
I20260807 02:57:06.695660 14748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:57:06.705272 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45561
--builtin_ntp_servers=127.1.121.84:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 02:57:07.083534 14762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:07.083859 14762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:07.084059 14762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:07.092382 14762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:07.092510 14762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:07.092725 14762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:57:07.103293 14762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:07.104813 14762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:07.106878 14762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:07.122795 14770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:07.122747 14773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:07.122721 14776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:07.124293 14762 server_base.cc:1034] running on GCE node
I20260807 02:57:07.125151 14762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:07.126452 14762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:07.127753 14762 hybrid_clock.cc:648] HybridClock initialized: now 1786071427127669 us; error 68 us; skew 500 ppm
I20260807 02:57:07.128142 14762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:57:07.134593 14762 webserver.cc:460] Webserver started at http://127.1.121.65:43697/ using document root <none> and password file <none>
I20260807 02:57:07.135434 14762 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:07.135581 14762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:07.135941 14762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:07.138932 14762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "a77f2c1ad12f499ba39bfd265a19ab68"
format_stamp: "Formatted at 2026-08-07 02:57:07 on dist-test-slave-3h5h"
server_key: "eec36c5ee5eee28929c4d2ca450ce15e"
server_key_iv: "a72d2a1ebf1e0f15bdd6327561dca886"
server_key_version: "encryptionkey@0"
I20260807 02:57:07.139765 14762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "a77f2c1ad12f499ba39bfd265a19ab68"
format_stamp: "Formatted at 2026-08-07 02:57:07 on dist-test-slave-3h5h"
server_key: "eec36c5ee5eee28929c4d2ca450ce15e"
server_key_iv: "a72d2a1ebf1e0f15bdd6327561dca886"
server_key_version: "encryptionkey@0"
I20260807 02:57:07.145592 14762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 02:57:07.149684 14784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:07.151074 14762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:57:07.151264 14762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "a77f2c1ad12f499ba39bfd265a19ab68"
format_stamp: "Formatted at 2026-08-07 02:57:07 on dist-test-slave-3h5h"
server_key: "eec36c5ee5eee28929c4d2ca450ce15e"
server_key_iv: "a72d2a1ebf1e0f15bdd6327561dca886"
server_key_version: "encryptionkey@0"
I20260807 02:57:07.151520 14762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:07.162976 14762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:07.163826 14762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:07.164147 14762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:07.165273 14762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:57:07.166759 14762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:57:07.166888 14762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:07.167021 14762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:57:07.167117 14762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:07.206157 14762 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:43017
I20260807 02:57:07.206269 14920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:43017 every 8 connection(s)
I20260807 02:57:07.207643 14762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:57:07.210189 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 14762
I20260807 02:57:07.210403 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:57:07.211347 1509 external_mini_cluster.cc:1428] Setting key c4e94674cfc4c8a303eef8e06f26cb74
I20260807 02:57:07.214967 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45561
--builtin_ntp_servers=127.1.121.84:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 02:57:07.223398 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:07.210620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:45445 (local address 127.1.121.126:45561)
0807 02:57:07.211170 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:07.211177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:07.212635 (+ 1458us) server_negotiation.cc:407] Connection header received
0807 02:57:07.215365 (+ 2730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:07.215374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:07.215471 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:07.215597 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:07.216753 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:07.218084 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:07.219832 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:07.220610 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:07.220778 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:07.222314 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:07.222324 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:07.222332 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:07.222361 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:07.222419 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:07.222440 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:07.222545 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:07.222776 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 02:57:07.222904 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":262,"threads_started":1}
I20260807 02:57:07.224259 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:07.211249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45561 (local address 127.1.121.65:45445)
0807 02:57:07.212446 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:07.212498 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:07.214887 (+ 2389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:07.215815 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:07.215823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:07.215919 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:07.216559 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:07.216577 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:07.218266 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:07.218270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:07.219656 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:07.219670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:07.220921 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:07.220928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:07.221153 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:07.221897 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:57:07.221900 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:07.221942 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:57:07.222090 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:57:07.222555 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:57:07.222565 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:57:07.222572 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 02:57:07.222724 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 02:57:07.223096 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":375,"threads_started":1}
I20260807 02:57:07.225972 14921 heartbeater.cc:347] Connected to a master server at 127.1.121.126:45561
I20260807 02:57:07.226354 14921 heartbeater.cc:464] Registering TS with master...
I20260807 02:57:07.227388 14921 heartbeater.cc:511] Master 127.1.121.126:45561 requested a full tablet report, sending...
I20260807 02:57:07.230100 14685 ts_manager.cc:194] Registered new tserver with Master: a77f2c1ad12f499ba39bfd265a19ab68 (127.1.121.65:43017)
I20260807 02:57:07.232374 14685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.65:45445
W20260807 02:57:07.629453 14925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:07.629779 14925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:07.629959 14925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:07.638535 14925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:07.638643 14925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:07.638823 14925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:57:07.649187 14925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:07.650642 14925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:07.652738 14925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:07.667551 14934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:07.667615 14933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:07.667611 14936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:07.668598 14925 server_base.cc:1034] running on GCE node
I20260807 02:57:07.669348 14925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:07.670315 14925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:07.671572 14925 hybrid_clock.cc:648] HybridClock initialized: now 1786071427671495 us; error 63 us; skew 500 ppm
I20260807 02:57:07.671984 14925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:57:07.674928 14925 webserver.cc:460] Webserver started at http://127.1.121.66:44221/ using document root <none> and password file <none>
I20260807 02:57:07.675917 14925 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:07.676091 14925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:07.676527 14925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:07.680101 14925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "2013761c02a7427f8b8b0167002c3b59"
format_stamp: "Formatted at 2026-08-07 02:57:07 on dist-test-slave-3h5h"
server_key: "f186dd55b756fb9f26d76f0c75a19fa3"
server_key_iv: "75570768318287069c2c3d90770efc4e"
server_key_version: "encryptionkey@0"
I20260807 02:57:07.681166 14925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "2013761c02a7427f8b8b0167002c3b59"
format_stamp: "Formatted at 2026-08-07 02:57:07 on dist-test-slave-3h5h"
server_key: "f186dd55b756fb9f26d76f0c75a19fa3"
server_key_iv: "75570768318287069c2c3d90770efc4e"
server_key_version: "encryptionkey@0"
I20260807 02:57:07.687255 14925 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 02:57:07.691157 14944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:07.692517 14925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:57:07.692718 14925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "2013761c02a7427f8b8b0167002c3b59"
format_stamp: "Formatted at 2026-08-07 02:57:07 on dist-test-slave-3h5h"
server_key: "f186dd55b756fb9f26d76f0c75a19fa3"
server_key_iv: "75570768318287069c2c3d90770efc4e"
server_key_version: "encryptionkey@0"
I20260807 02:57:07.693071 14925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:07.711110 14925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:07.711939 14925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:07.712267 14925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:07.713310 14925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:57:07.714869 14925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:57:07.715070 14925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:07.715207 14925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:57:07.715298 14925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:07.750592 14925 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:37789
I20260807 02:57:07.750641 15081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:37789 every 8 connection(s)
I20260807 02:57:07.751984 14925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:57:07.753697 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 14925
I20260807 02:57:07.753870 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:57:07.754323 1509 external_mini_cluster.cc:1428] Setting key dbacf77f9d7cd1b50cfd45265f8bb589
I20260807 02:57:07.757668 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45561
--builtin_ntp_servers=127.1.121.84:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 02:57:07.766850 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:07.755037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:55313 (local address 127.1.121.126:45561)
0807 02:57:07.757233 (+ 2196us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:07.757239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:07.758114 (+ 875us) server_negotiation.cc:407] Connection header received
0807 02:57:07.759847 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:07.759861 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:07.759955 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:07.760100 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:07.761248 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:07.762385 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:07.763949 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:07.764377 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:07.764524 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:07.765981 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:07.765986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:07.765990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:07.766008 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:07.766048 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:07.766060 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:07.766170 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:07.766400 (+ 230us) server_negotiation.cc:299] Negotiation successful
0807 02:57:07.766499 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2068,"thread_start_us":1958,"threads_started":1}
I20260807 02:57:07.767923 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:07.755967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45561 (local address 127.1.121.66:55313)
0807 02:57:07.757939 (+ 1972us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:07.757984 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:07.759563 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:07.760297 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:07.760306 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:07.760393 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:07.761072 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:07.761093 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:07.762549 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:07.762554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:07.763795 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:07.763812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:07.764628 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:07.764636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:07.764925 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:07.765653 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:57:07.765656 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:07.765695 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:57:07.765771 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:57:07.766169 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:57:07.766183 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:57:07.766194 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 02:57:07.766369 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 02:57:07.766779 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":1152,"threads_started":1}
I20260807 02:57:07.769441 15082 heartbeater.cc:347] Connected to a master server at 127.1.121.126:45561
I20260807 02:57:07.769824 15082 heartbeater.cc:464] Registering TS with master...
I20260807 02:57:07.770799 15082 heartbeater.cc:511] Master 127.1.121.126:45561 requested a full tablet report, sending...
I20260807 02:57:07.772379 14685 ts_manager.cc:194] Registered new tserver with Master: 2013761c02a7427f8b8b0167002c3b59 (127.1.121.66:37789)
I20260807 02:57:07.773722 14685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.66:55313
W20260807 02:57:08.123045 15087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:08.123337 15087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:08.123436 15087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:08.130942 15087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:08.131052 15087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:08.131165 15087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:57:08.140530 15087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:45561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:08.141964 15087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:08.143810 15087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:08.159106 15097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:08.159106 15099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:08.159257 15095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:08.159863 15087 server_base.cc:1034] running on GCE node
I20260807 02:57:08.160559 15087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:08.161616 15087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:08.162819 15087 hybrid_clock.cc:648] HybridClock initialized: now 1786071428162759 us; error 61 us; skew 500 ppm
I20260807 02:57:08.163166 15087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:57:08.165791 15087 webserver.cc:460] Webserver started at http://127.1.121.67:45319/ using document root <none> and password file <none>
I20260807 02:57:08.166530 15087 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:08.166625 15087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:08.166990 15087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:08.170074 15087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "d2e09ae587d84beeae72e3d794897359"
format_stamp: "Formatted at 2026-08-07 02:57:08 on dist-test-slave-3h5h"
server_key: "f4debaccd453934b40da8f3af9f2d63c"
server_key_iv: "14af99dd87dcb56864ae38047352d72d"
server_key_version: "encryptionkey@0"
I20260807 02:57:08.170902 15087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "d2e09ae587d84beeae72e3d794897359"
format_stamp: "Formatted at 2026-08-07 02:57:08 on dist-test-slave-3h5h"
server_key: "f4debaccd453934b40da8f3af9f2d63c"
server_key_iv: "14af99dd87dcb56864ae38047352d72d"
server_key_version: "encryptionkey@0"
I20260807 02:57:08.176935 15087 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 02:57:08.180620 15105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:08.182106 15087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 02:57:08.182317 15087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "d2e09ae587d84beeae72e3d794897359"
format_stamp: "Formatted at 2026-08-07 02:57:08 on dist-test-slave-3h5h"
server_key: "f4debaccd453934b40da8f3af9f2d63c"
server_key_iv: "14af99dd87dcb56864ae38047352d72d"
server_key_version: "encryptionkey@0"
I20260807 02:57:08.182544 15087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:08.196412 15087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:08.197449 15087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:08.197806 15087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:08.198781 15087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:57:08.200222 15087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:57:08.200344 15087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:08.200479 15087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:57:08.200573 15087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:08.235803 14921 heartbeater.cc:503] Master 127.1.121.126:45561 was elected leader, sending a full tablet report...
I20260807 02:57:08.235913 15087 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:43751
I20260807 02:57:08.235961 15257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:43751 every 8 connection(s)
I20260807 02:57:08.237370 15087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:57:08.240447 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 15087
I20260807 02:57:08.240614 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:57:08.241293 1509 external_mini_cluster.cc:1428] Setting key def490e6fe79b9616af0a510d3d8fc16
I20260807 02:57:08.250824 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:08.240022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:42317 (local address 127.1.121.126:45561)
0807 02:57:08.240281 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:08.240288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:08.242045 (+ 1757us) server_negotiation.cc:407] Connection header received
0807 02:57:08.244326 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:08.244331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:08.244395 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:08.244513 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:08.245705 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.246522 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.248175 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.248588 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.248694 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:08.250101 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:08.250107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:08.250111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:08.250131 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:08.250166 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:08.250179 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:08.250268 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:08.250506 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 02:57:08.250592 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 02:57:08.251726 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:08.240635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:45561 (local address 127.1.121.67:42317)
0807 02:57:08.241872 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:08.241929 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:08.244022 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:08.244748 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:08.244755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:08.244884 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:08.245517 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:08.245558 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.246674 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.246678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:08.248031 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:08.248044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.248786 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.248790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:08.249039 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:08.249718 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:57:08.249720 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:08.249758 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:57:08.249831 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:57:08.250278 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:57:08.250286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:57:08.250291 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 02:57:08.250429 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 02:57:08.250704 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":396,"threads_started":1}
I20260807 02:57:08.253239 15259 heartbeater.cc:347] Connected to a master server at 127.1.121.126:45561
I20260807 02:57:08.253535 15259 heartbeater.cc:464] Registering TS with master...
I20260807 02:57:08.254350 15259 heartbeater.cc:511] Master 127.1.121.126:45561 requested a full tablet report, sending...
I20260807 02:57:08.256040 14685 ts_manager.cc:194] Registered new tserver with Master: d2e09ae587d84beeae72e3d794897359 (127.1.121.67:43751)
I20260807 02:57:08.257087 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:57:08.257194 14685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.67:42317
W20260807 02:57:08.264235 15085 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:08.262466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.1.121.126:45561)
0807 02:57:08.262642 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:08.262648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:08.263094 (+ 446us) server_negotiation.cc:407] Connection header received
0807 02:57:08.263289 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:08.263295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:08.263369 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:08.263473 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:08.263945 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36342: BlockingRecv error: recv got EOF from 127.0.0.1:36342 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:57:08.265182 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 14762
I20260807 02:57:08.273451 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:08.266294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.1.121.65:43017)
0807 02:57:08.266881 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:08.266888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:08.266915 (+ 27us) server_negotiation.cc:407] Connection header received
0807 02:57:08.266971 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:08.266977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:08.267181 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:08.267325 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:08.267759 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.269018 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.270284 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.270818 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.270913 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:08.271945 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:08.271954 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:08.271967 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:08.272550 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:08.272602 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:08.272622 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:08.272714 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:08.272923 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 02:57:08.273065 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":275,"threads_started":1}
I20260807 02:57:08.274715 14916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:08.358754 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 14762
I20260807 02:57:08.382328 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 14925
I20260807 02:57:08.390633 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:08.383376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.1.121.66:37789)
0807 02:57:08.383997 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:08.384004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:08.384033 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:57:08.384107 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:08.384113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:08.384358 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:08.384489 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:08.384995 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.386021 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.387330 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.387868 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.387972 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:08.389198 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:08.389207 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:08.389220 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:08.389825 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:08.389871 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:08.389890 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:08.389979 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:08.390175 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 02:57:08.390312 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":342,"threads_started":1}
I20260807 02:57:08.391927 15076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:08.476706 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 14925
I20260807 02:57:08.501384 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 15087
I20260807 02:57:08.510214 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:08.502508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.1.121.67:43751)
0807 02:57:08.503135 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:08.503143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:08.503175 (+ 32us) server_negotiation.cc:407] Connection header received
0807 02:57:08.503238 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:08.503245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:08.503469 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:08.503697 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:08.504196 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.505383 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.506689 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:08.507282 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:08.507441 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:08.508548 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:08.508557 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:08.508573 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:08.509288 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:08.509337 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:08.509358 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:08.509509 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:08.509664 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 02:57:08.509809 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":310,"threads_started":1}
I20260807 02:57:08.511643 15252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:08.595487 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 15087
I20260807 02:57:08.623642 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 14643
I20260807 02:57:08.625187 14723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:08.731137 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 14643
2026-08-07T02:57:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2718 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T02:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:57:08Z Disabled control of system clock
I20260807 02:57:08.829722 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:38805
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:38805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 02:57:09.198871 15286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:09.199186 15286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:09.199404 15286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:09.206909 15286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:57:09.207043 15286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:09.207083 15286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:09.207203 15286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:57:09.207276 15286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:57:09.216740 15286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:38805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:38805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:09.218216 15286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:09.220368 15286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:09.233417 15293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:09.233441 15292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:09.234131 15296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:09.234450 15286 server_base.cc:1034] running on GCE node
I20260807 02:57:09.235263 15286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:09.236759 15286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:09.238045 15286 hybrid_clock.cc:648] HybridClock initialized: now 1786071429237981 us; error 49 us; skew 500 ppm
I20260807 02:57:09.238458 15286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:57:09.241456 15286 webserver.cc:460] Webserver started at http://127.1.121.126:37765/ using document root <none> and password file <none>
I20260807 02:57:09.242250 15286 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:09.242389 15286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:09.242750 15286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:09.245935 15286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "8ae23e7c4fe344e2abe8caa81dd0905e"
format_stamp: "Formatted at 2026-08-07 02:57:09 on dist-test-slave-3h5h"
server_key: "79524184c78895209431d54cba3afd6d"
server_key_iv: "57cc33caadcb4bf4d1b319485194308e"
server_key_version: "encryptionkey@0"
I20260807 02:57:09.246822 15286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "8ae23e7c4fe344e2abe8caa81dd0905e"
format_stamp: "Formatted at 2026-08-07 02:57:09 on dist-test-slave-3h5h"
server_key: "79524184c78895209431d54cba3afd6d"
server_key_iv: "57cc33caadcb4bf4d1b319485194308e"
server_key_version: "encryptionkey@0"
I20260807 02:57:09.252435 15286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 02:57:09.256281 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:09.257910 15286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:57:09.258117 15286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "8ae23e7c4fe344e2abe8caa81dd0905e"
format_stamp: "Formatted at 2026-08-07 02:57:09 on dist-test-slave-3h5h"
server_key: "79524184c78895209431d54cba3afd6d"
server_key_iv: "57cc33caadcb4bf4d1b319485194308e"
server_key_version: "encryptionkey@0"
I20260807 02:57:09.258378 15286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:09.264325 15286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:09.265254 15286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:09.265573 15286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:09.286060 15286 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:38805
I20260807 02:57:09.286191 15376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:38805 every 8 connection(s)
I20260807 02:57:09.287662 15286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:57:09.290400 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 15286
I20260807 02:57:09.290665 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:57:09.291579 1509 external_mini_cluster.cc:1428] Setting key 53786baeeda2bf0abe1bff669010d747
I20260807 02:57:09.292928 15377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:57:09.302605 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:09.293987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.1.121.126:38805)
0807 02:57:09.295106 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:09.295117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:09.295161 (+ 44us) server_negotiation.cc:407] Connection header received
0807 02:57:09.297505 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:09.297549 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:09.298232 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:09.298778 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:09.300230 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:09.300239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:09.300278 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:09.300299 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:09.300421 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:09.300465 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:09.300721 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:09.300911 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 02:57:09.301233 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":416,"threads_started":1}
I20260807 02:57:09.310427 15377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e: Bootstrap starting.
I20260807 02:57:09.313952 15377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e: Neither blocks nor log segments found. Creating new log.
I20260807 02:57:09.315105 15377 log.cc:824] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:57:09.318360 15377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e: No bootstrap required, opened a new log
I20260807 02:57:09.326395 15377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae23e7c4fe344e2abe8caa81dd0905e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38805 } }
I20260807 02:57:09.326815 15377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:57:09.326938 15377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae23e7c4fe344e2abe8caa81dd0905e, State: Initialized, Role: FOLLOWER
I20260807 02:57:09.327582 15377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae23e7c4fe344e2abe8caa81dd0905e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38805 } }
I20260807 02:57:09.327865 15377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:57:09.328034 15377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:57:09.328224 15377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:57:09.330925 15377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae23e7c4fe344e2abe8caa81dd0905e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38805 } }
I20260807 02:57:09.331482 15377 leader_election.cc:302] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ae23e7c4fe344e2abe8caa81dd0905e; no voters:
I20260807 02:57:09.332247 15377 leader_election.cc:288] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:57:09.332391 15383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:57:09.334610 15383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [term 1 LEADER]: Becoming Leader. State: Replica: 8ae23e7c4fe344e2abe8caa81dd0905e, State: Running, Role: LEADER
I20260807 02:57:09.335170 15377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:57:09.335220 15383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae23e7c4fe344e2abe8caa81dd0905e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38805 } }
I20260807 02:57:09.338696 15384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ae23e7c4fe344e2abe8caa81dd0905e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae23e7c4fe344e2abe8caa81dd0905e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38805 } } }
I20260807 02:57:09.339148 15384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [sys.catalog]: This master's current role is: LEADER
I20260807 02:57:09.338723 15385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ae23e7c4fe344e2abe8caa81dd0905e. Latest consensus state: current_term: 1 leader_uuid: "8ae23e7c4fe344e2abe8caa81dd0905e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae23e7c4fe344e2abe8caa81dd0905e" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 38805 } } }
I20260807 02:57:09.340488 15385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e [sys.catalog]: This master's current role is: LEADER
I20260807 02:57:09.340806 15396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:57:09.347414 15396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:57:09.356657 15396 catalog_manager.cc:1347] Generated new cluster ID: 25a10ea5ad374fa28268ec2e31f239a7
I20260807 02:57:09.356789 15396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:57:09.370271 15396 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:57:09.370756 15396 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:57:09.379930 15396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ae23e7c4fe344e2abe8caa81dd0905e: Generated new TSK 0
I20260807 02:57:09.380878 15396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:57:09.392196 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38805
--builtin_ntp_servers=127.1.121.84:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 02:57:09.754271 15406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:09.754547 15406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:09.754715 15406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:09.762216 15406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:09.762315 15406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:09.762481 15406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:57:09.772047 15406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:09.773502 15406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:09.775331 15406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:09.789196 15413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:09.789273 15412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:09.789204 15416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:09.793998 15406 server_base.cc:1034] running on GCE node
I20260807 02:57:09.794629 15406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:09.795641 15406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:09.796936 15406 hybrid_clock.cc:648] HybridClock initialized: now 1786071429796872 us; error 46 us; skew 500 ppm
I20260807 02:57:09.797281 15406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:57:09.799860 15406 webserver.cc:460] Webserver started at http://127.1.121.65:36223/ using document root <none> and password file <none>
I20260807 02:57:09.800617 15406 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:09.800756 15406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:09.801133 15406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:09.804171 15406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "2e399d7cb0714afea37da5676d74d3b2"
format_stamp: "Formatted at 2026-08-07 02:57:09 on dist-test-slave-3h5h"
server_key: "76deb4ca5727b0340956253b532f59c1"
server_key_iv: "b14a117a308def58308b37e9a3b0b904"
server_key_version: "encryptionkey@0"
I20260807 02:57:09.805051 15406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "2e399d7cb0714afea37da5676d74d3b2"
format_stamp: "Formatted at 2026-08-07 02:57:09 on dist-test-slave-3h5h"
server_key: "76deb4ca5727b0340956253b532f59c1"
server_key_iv: "b14a117a308def58308b37e9a3b0b904"
server_key_version: "encryptionkey@0"
I20260807 02:57:09.810273 15406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 02:57:09.813820 15427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:09.815331 15406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:57:09.815511 15406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "2e399d7cb0714afea37da5676d74d3b2"
format_stamp: "Formatted at 2026-08-07 02:57:09 on dist-test-slave-3h5h"
server_key: "76deb4ca5727b0340956253b532f59c1"
server_key_iv: "b14a117a308def58308b37e9a3b0b904"
server_key_version: "encryptionkey@0"
I20260807 02:57:09.815757 15406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:09.821744 15406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:09.822548 15406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:09.822888 15406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:09.823848 15406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:57:09.825552 15406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:57:09.825700 15406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:09.825836 15406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:57:09.825925 15406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:09.862054 15406 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:34667
I20260807 02:57:09.862149 15592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:34667 every 8 connection(s)
I20260807 02:57:09.863528 15406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:57:09.864135 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 15406
I20260807 02:57:09.864426 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:57:09.865592 1509 external_mini_cluster.cc:1428] Setting key 5cf49ee07d0d9a1e237c0f11790573eb
I20260807 02:57:09.869356 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38805
--builtin_ntp_servers=127.1.121.84:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 02:57:09.874617 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:09.866556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:53433 (local address 127.1.121.126:38805)
0807 02:57:09.867137 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:09.867142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:09.868689 (+ 1547us) server_negotiation.cc:407] Connection header received
0807 02:57:09.871901 (+ 3212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:09.871906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:09.871959 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:09.872054 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:09.873890 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:09.873895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:09.873898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:09.873914 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:09.873955 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:09.873967 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:09.874063 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:09.874332 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 02:57:09.874384 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":326,"threads_started":1}
I20260807 02:57:09.875782 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:09.867195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:38805 (local address 127.1.121.65:53433)
0807 02:57:09.868512 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:09.868566 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:09.871557 (+ 2991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:09.872385 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:09.872401 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:09.872555 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:09.873441 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:57:09.873448 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:09.873511 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:57:09.873647 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:57:09.874077 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:57:09.874092 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:57:09.874104 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 02:57:09.874448 (+ 344us) client_negotiation.cc:240] Negotiation successful
0807 02:57:09.874741 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":403,"threads_started":1}
I20260807 02:57:09.877600 15594 heartbeater.cc:347] Connected to a master server at 127.1.121.126:38805
I20260807 02:57:09.877930 15594 heartbeater.cc:464] Registering TS with master...
I20260807 02:57:09.878569 15594 heartbeater.cc:511] Master 127.1.121.126:38805 requested a full tablet report, sending...
I20260807 02:57:09.881121 15327 ts_manager.cc:194] Registered new tserver with Master: 2e399d7cb0714afea37da5676d74d3b2 (127.1.121.65:34667)
W20260807 02:57:10.226219 15602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:10.226574 15602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:10.226758 15602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:10.234586 15602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:10.234738 15602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:10.234946 15602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:57:10.244722 15602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:10.246130 15602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:10.248029 15602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:10.262039 15614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:10.262040 15608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:10.262112 15612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:10.262910 15602 server_base.cc:1034] running on GCE node
I20260807 02:57:10.263563 15602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:10.264600 15602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:10.265807 15602 hybrid_clock.cc:648] HybridClock initialized: now 1786071430265728 us; error 78 us; skew 500 ppm
I20260807 02:57:10.266148 15602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:57:10.268481 15602 webserver.cc:460] Webserver started at http://127.1.121.66:43969/ using document root <none> and password file <none>
I20260807 02:57:10.269263 15602 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:10.269356 15602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:10.269699 15602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:10.272747 15602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "d30f389224f446a3928af872d1811869"
format_stamp: "Formatted at 2026-08-07 02:57:10 on dist-test-slave-3h5h"
server_key: "1ab84d9a51343fd5d32d59530d1ba205"
server_key_iv: "11f0d90e0e0965c57d54d177ee5b6fdc"
server_key_version: "encryptionkey@0"
I20260807 02:57:10.273634 15602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "d30f389224f446a3928af872d1811869"
format_stamp: "Formatted at 2026-08-07 02:57:10 on dist-test-slave-3h5h"
server_key: "1ab84d9a51343fd5d32d59530d1ba205"
server_key_iv: "11f0d90e0e0965c57d54d177ee5b6fdc"
server_key_version: "encryptionkey@0"
I20260807 02:57:10.278995 15602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:57:10.282706 15623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:10.284163 15602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 02:57:10.284298 15602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "d30f389224f446a3928af872d1811869"
format_stamp: "Formatted at 2026-08-07 02:57:10 on dist-test-slave-3h5h"
server_key: "1ab84d9a51343fd5d32d59530d1ba205"
server_key_iv: "11f0d90e0e0965c57d54d177ee5b6fdc"
server_key_version: "encryptionkey@0"
I20260807 02:57:10.284492 15602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:10.292483 15602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:10.293598 15602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:10.293915 15602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:10.295053 15602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:57:10.297070 15602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:57:10.297174 15602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:10.297268 15602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:57:10.297338 15602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:10.336504 15602 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:33863
I20260807 02:57:10.336604 15771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:33863 every 8 connection(s)
I20260807 02:57:10.338063 15602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:57:10.342464 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 15602
I20260807 02:57:10.342661 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:57:10.343361 1509 external_mini_cluster.cc:1428] Setting key 309267b07b1e15fff90773792731882f
I20260807 02:57:10.346992 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38805
--builtin_ntp_servers=127.1.121.84:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 02:57:10.347936 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:10.341058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:44727 (local address 127.1.121.126:38805)
0807 02:57:10.341252 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:10.341256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:10.343109 (+ 1853us) server_negotiation.cc:407] Connection header received
0807 02:57:10.345228 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:10.345235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:10.345354 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:10.345490 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:10.347166 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:10.347175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:10.347184 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:10.347221 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:10.347285 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:10.347300 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:10.347403 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:10.347649 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 02:57:10.347706 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 02:57:10.349207 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:10.341683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:38805 (local address 127.1.121.66:44727)
0807 02:57:10.342891 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:10.342951 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:10.344957 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:10.345787 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:10.345800 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:10.345918 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:10.346700 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:57:10.346706 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:10.346760 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:57:10.346875 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:57:10.347409 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:57:10.347423 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:57:10.347434 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 02:57:10.347720 (+ 286us) client_negotiation.cc:240] Negotiation successful
0807 02:57:10.347968 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":350,"threads_started":1}
I20260807 02:57:10.350684 15772 heartbeater.cc:347] Connected to a master server at 127.1.121.126:38805
I20260807 02:57:10.350999 15772 heartbeater.cc:464] Registering TS with master...
I20260807 02:57:10.351717 15772 heartbeater.cc:511] Master 127.1.121.126:38805 requested a full tablet report, sending...
I20260807 02:57:10.353577 15327 ts_manager.cc:194] Registered new tserver with Master: d30f389224f446a3928af872d1811869 (127.1.121.66:33863)
W20260807 02:57:10.703329 15775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:10.703644 15775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:10.703758 15775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:10.711017 15775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:10.711125 15775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:10.711236 15775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:57:10.720674 15775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:38805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:10.722146 15775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:10.723884 15775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:10.737959 15784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:10.737928 15789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:10.737986 15785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:10.742571 15775 server_base.cc:1034] running on GCE node
I20260807 02:57:10.743196 15775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:10.744163 15775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:10.745368 15775 hybrid_clock.cc:648] HybridClock initialized: now 1786071430745321 us; error 33 us; skew 500 ppm
I20260807 02:57:10.745666 15775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 02:57:10.748070 15775 webserver.cc:460] Webserver started at http://127.1.121.67:33977/ using document root <none> and password file <none>
I20260807 02:57:10.748739 15775 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:10.748824 15775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:10.749212 15775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:10.752084 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "9f7ea707db1c43e699abecfd91259f8f"
format_stamp: "Formatted at 2026-08-07 02:57:10 on dist-test-slave-3h5h"
server_key: "f0564f1efe403c6caf5a28e449c1340d"
server_key_iv: "1e71e0417a73e761481207992236d758"
server_key_version: "encryptionkey@0"
I20260807 02:57:10.752825 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "9f7ea707db1c43e699abecfd91259f8f"
format_stamp: "Formatted at 2026-08-07 02:57:10 on dist-test-slave-3h5h"
server_key: "f0564f1efe403c6caf5a28e449c1340d"
server_key_iv: "1e71e0417a73e761481207992236d758"
server_key_version: "encryptionkey@0"
I20260807 02:57:10.758042 15775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 02:57:10.761739 15796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:10.763181 15775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:57:10.763370 15775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "9f7ea707db1c43e699abecfd91259f8f"
format_stamp: "Formatted at 2026-08-07 02:57:10 on dist-test-slave-3h5h"
server_key: "f0564f1efe403c6caf5a28e449c1340d"
server_key_iv: "1e71e0417a73e761481207992236d758"
server_key_version: "encryptionkey@0"
I20260807 02:57:10.763604 15775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:10.769579 15775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:10.770402 15775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:10.770692 15775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:10.771651 15775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:57:10.773201 15775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:57:10.773325 15775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:10.773470 15775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:57:10.773558 15775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:10.810518 15775 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:37631
I20260807 02:57:10.810611 15935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:37631 every 8 connection(s)
I20260807 02:57:10.812045 15775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:57:10.818701 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 15775
I20260807 02:57:10.818934 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:57:10.819554 1509 external_mini_cluster.cc:1428] Setting key da7c6534d46a1646857002ce63eb1e27
I20260807 02:57:10.821017 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:10.814771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:39967 (local address 127.1.121.126:38805)
0807 02:57:10.814998 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:10.815005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:10.816493 (+ 1488us) server_negotiation.cc:407] Connection header received
0807 02:57:10.818261 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:10.818266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:10.818333 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:10.818425 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:10.820064 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:10.820070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:10.820075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:10.820096 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:10.820156 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:10.820176 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:10.820310 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:10.820534 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 02:57:10.820636 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 02:57:10.822036 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:10.815414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:38805 (local address 127.1.121.67:39967)
0807 02:57:10.816331 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:10.816366 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:10.817989 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:10.818733 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:10.818747 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:10.818872 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:10.819672 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:57:10.819676 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:10.819728 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:57:10.819845 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:57:10.820291 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:57:10.820300 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:57:10.820306 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 02:57:10.820607 (+ 301us) client_negotiation.cc:240] Negotiation successful
0807 02:57:10.820899 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":276,"threads_started":1}
I20260807 02:57:10.823433 15936 heartbeater.cc:347] Connected to a master server at 127.1.121.126:38805
I20260807 02:57:10.823746 15936 heartbeater.cc:464] Registering TS with master...
I20260807 02:57:10.824355 15936 heartbeater.cc:511] Master 127.1.121.126:38805 requested a full tablet report, sending...
I20260807 02:57:10.825842 15326 ts_manager.cc:194] Registered new tserver with Master: 9f7ea707db1c43e699abecfd91259f8f (127.1.121.67:37631)
I20260807 02:57:10.835815 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 02:57:10.842371 15597 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:57:10.840608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.1.121.126:38805)
0807 02:57:10.840805 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:10.840812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:10.841355 (+ 543us) server_negotiation.cc:407] Connection header received
0807 02:57:10.841558 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:10.841564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:10.841631 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:10.841738 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:10.842088 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57564: BlockingRecv error: recv got EOF from 127.0.0.1:57564 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:57:10.843307 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 15406
I20260807 02:57:10.847079 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:10.844331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.1.121.65:34667)
0807 02:57:10.844924 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:10.844931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:10.844957 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:57:10.845017 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:10.845022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:10.845240 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:10.845367 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:10.846251 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:10.846257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:10.846268 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:10.846285 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:10.846324 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:10.846337 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:10.846445 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:10.846649 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 02:57:10.846765 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":278,"threads_started":1}
I20260807 02:57:10.848416 15587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:10.932825 15594 heartbeater.cc:503] Master 127.1.121.126:38805 was elected leader, sending a full tablet report...
I20260807 02:57:10.934624 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 15406
W20260807 02:57:10.960062 15305 connection.cc:535] server connection from 127.1.121.65:53433 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 02:57:10.960245 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 15602
I20260807 02:57:10.964602 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:10.961398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.1.121.66:33863)
0807 02:57:10.961959 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:10.961967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:10.961996 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:57:10.962060 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:10.962073 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:10.962299 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:10.962429 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:10.963678 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:10.963686 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:10.963699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:10.963720 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:10.963768 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:10.963783 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:10.963891 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:10.964199 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 02:57:10.964334 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":266,"threads_started":1}
I20260807 02:57:10.966132 15766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:11.059055 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 15602
I20260807 02:57:11.084244 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 15775
I20260807 02:57:11.088757 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:11.085596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.1.121.67:37631)
0807 02:57:11.086300 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:11.086307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:11.086337 (+ 30us) server_negotiation.cc:407] Connection header received
0807 02:57:11.086401 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:11.086408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:11.086681 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:11.086812 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:11.087851 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:11.087857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:11.087868 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:57:11.087885 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:11.087933 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:57:11.087948 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:11.088083 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:11.088373 (+ 290us) server_negotiation.cc:299] Negotiation successful
0807 02:57:11.088455 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":378,"threads_started":1}
I20260807 02:57:11.090130 15929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:11.179315 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 15775
I20260807 02:57:11.204495 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 15286
I20260807 02:57:11.206139 15370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:11.315403 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 15286
2026-08-07T02:57:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2583 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:57:11 dist-test-slave-3h5h krb5kdc[15960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:57:11 dist-test-slave-3h5h krb5kdc[15960](info): set up 2 sockets
Aug 07 02:57:11 dist-test-slave-3h5h krb5kdc[15960](info): commencing operation
krb5kdc: starting...
W20260807 02:57:12.695078 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.327s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:57:12 dist-test-slave-3h5h krb5kdc[15960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:57:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:57:13.029269 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33251
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:33251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:57:13.395098 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:13.395457 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:13.395627 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:13.402912 15979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:57:13.403064 15979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:13.403177 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:13.403282 15979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:57:13.403371 15979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:57:13.413286 15979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:33251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:13.414757 15979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:13.416239 15979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:13.428356 15985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:13.428248 15986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:13.428293 15988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:13.429819 15979 server_base.cc:1034] running on GCE node
I20260807 02:57:13.430624 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:13.432036 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:13.433252 15979 hybrid_clock.cc:648] HybridClock initialized: now 1786071433433209 us; error 32 us; skew 500 ppm
I20260807 02:57:13.433562 15979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:57:13 dist-test-slave-3h5h krb5kdc[15960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071433, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:57:13.441890 15979 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:57:13.443473 15979 webserver.cc:460] Webserver started at http://127.1.121.126:36035/ using document root <none> and password file <none>
I20260807 02:57:13.444169 15979 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:13.444281 15979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:13.444540 15979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:13.447654 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "3106995c61cd42b285375b83e240bf32"
format_stamp: "Formatted at 2026-08-07 02:57:13 on dist-test-slave-3h5h"
server_key: "4afe7af9ab083db753a42d2300f1fd81"
server_key_iv: "b38eb6d2a26b82c73c23b6b9d5418fc1"
server_key_version: "encryptionkey@0"
I20260807 02:57:13.448369 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "3106995c61cd42b285375b83e240bf32"
format_stamp: "Formatted at 2026-08-07 02:57:13 on dist-test-slave-3h5h"
server_key: "4afe7af9ab083db753a42d2300f1fd81"
server_key_iv: "b38eb6d2a26b82c73c23b6b9d5418fc1"
server_key_version: "encryptionkey@0"
I20260807 02:57:13.453634 15979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 02:57:13.457144 15998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:13.458477 15979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:57:13.458675 15979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "3106995c61cd42b285375b83e240bf32"
format_stamp: "Formatted at 2026-08-07 02:57:13 on dist-test-slave-3h5h"
server_key: "4afe7af9ab083db753a42d2300f1fd81"
server_key_iv: "b38eb6d2a26b82c73c23b6b9d5418fc1"
server_key_version: "encryptionkey@0"
I20260807 02:57:13.458873 15979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:13.470094 15979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:13.474216 15979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:13.474503 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:13.504707 15979 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:33251
I20260807 02:57:13.504745 16066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:33251 every 8 connection(s)
I20260807 02:57:13.506237 15979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:57:13.510782 16067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:57:13.511075 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 15979
I20260807 02:57:13.511361 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:57:13.512427 1509 external_mini_cluster.cc:1428] Setting key 60d450d38122179d798e07092adbd7ab
I20260807 02:57:13.528316 16067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32: Bootstrap starting.
Aug 07 02:57:13 dist-test-slave-3h5h krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:57:13.532385 16067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32: Neither blocks nor log segments found. Creating new log.
I20260807 02:57:13.533762 16067 log.cc:824] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32: Log is configured to *not* fsync() on all Append() calls
I20260807 02:57:13.537259 16067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32: No bootstrap required, opened a new log
I20260807 02:57:13.538380 16070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:13.514894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.1.121.126:33251)
0807 02:57:13.515748 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:13.515759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:13.515804 (+ 45us) server_negotiation.cc:407] Connection header received
0807 02:57:13.518232 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:13.518255 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:13.518652 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:13.519000 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:13.521363 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:13.522647 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:13.524754 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:13.525530 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:13.525662 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:13.531010 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:13.531017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:13.531033 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:13.531077 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:13.534288 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:13.534973 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:13.534980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:13.534986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:13.535163 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:13.535558 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:13.535564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:13.535567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:13.536084 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:13.536421 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:13.536915 (+ 494us) server_negotiation.cc:299] Negotiation successful
0807 02:57:13.537376 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":400,"threads_started":1}
I20260807 02:57:13.547753 16067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3106995c61cd42b285375b83e240bf32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33251 } }
I20260807 02:57:13.548175 16067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:57:13.548314 16067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3106995c61cd42b285375b83e240bf32, State: Initialized, Role: FOLLOWER
I20260807 02:57:13.549067 16067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3106995c61cd42b285375b83e240bf32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33251 } }
I20260807 02:57:13.549324 16067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:57:13.549479 16067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:57:13.549664 16067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:57:13.552470 16067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3106995c61cd42b285375b83e240bf32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33251 } }
I20260807 02:57:13.553054 16067 leader_election.cc:302] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3106995c61cd42b285375b83e240bf32; no voters:
I20260807 02:57:13.553668 16067 leader_election.cc:288] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:57:13.553841 16073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:57:13.554585 16073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [term 1 LEADER]: Becoming Leader. State: Replica: 3106995c61cd42b285375b83e240bf32, State: Running, Role: LEADER
I20260807 02:57:13.555114 16073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3106995c61cd42b285375b83e240bf32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33251 } }
I20260807 02:57:13.556064 16067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:57:13.557971 16075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3106995c61cd42b285375b83e240bf32. Latest consensus state: current_term: 1 leader_uuid: "3106995c61cd42b285375b83e240bf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3106995c61cd42b285375b83e240bf32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33251 } } }
I20260807 02:57:13.558337 16075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [sys.catalog]: This master's current role is: LEADER
I20260807 02:57:13.558777 16074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3106995c61cd42b285375b83e240bf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3106995c61cd42b285375b83e240bf32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33251 } } }
I20260807 02:57:13.559232 16074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32 [sys.catalog]: This master's current role is: LEADER
I20260807 02:57:13.566001 16078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:57:13.570796 16078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:57:13.579432 16078 catalog_manager.cc:1347] Generated new cluster ID: 3155757b710d43cfb44882b68464dc8b
I20260807 02:57:13.579566 16078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:57:13.586300 16078 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:57:13.588272 16078 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:57:13.598915 16078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32: Generated new TSK 0
I20260807 02:57:13.599781 16078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:57:13.677506 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33251
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:57:14.048343 16098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:14.048722 16098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:14.048959 16098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:14.058077 16098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:14.058256 16098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:14.058477 16098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:57:14.070603 16098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:14.072134 16098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:14.073756 16098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:14.088712 16106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:14.088758 16109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:14.088678 16107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:14.089927 16098 server_base.cc:1034] running on GCE node
I20260807 02:57:14.090715 16098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:14.091873 16098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:14.093168 16098 hybrid_clock.cc:648] HybridClock initialized: now 1786071434093111 us; error 61 us; skew 500 ppm
I20260807 02:57:14.093567 16098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:57:14 dist-test-slave-3h5h krb5kdc[15960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071434, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:57:14.097322 16098 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:57:14.099179 16098 webserver.cc:460] Webserver started at http://127.1.121.65:43749/ using document root <none> and password file <none>
I20260807 02:57:14.100020 16098 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:14.100190 16098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:14.100541 16098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:14.104095 16098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "a69a0b67b8544c1bbf866921cdbe4306"
format_stamp: "Formatted at 2026-08-07 02:57:14 on dist-test-slave-3h5h"
server_key: "c8088b594fbc8dff6b0b350411d9641f"
server_key_iv: "c19e294817e0c71c642c2c66ccfdb1a4"
server_key_version: "encryptionkey@0"
I20260807 02:57:14.105077 16098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "a69a0b67b8544c1bbf866921cdbe4306"
format_stamp: "Formatted at 2026-08-07 02:57:14 on dist-test-slave-3h5h"
server_key: "c8088b594fbc8dff6b0b350411d9641f"
server_key_iv: "c19e294817e0c71c642c2c66ccfdb1a4"
server_key_version: "encryptionkey@0"
I20260807 02:57:14.110591 16098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:57:14.114410 16116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:14.115818 16098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 02:57:14.116014 16098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "a69a0b67b8544c1bbf866921cdbe4306"
format_stamp: "Formatted at 2026-08-07 02:57:14 on dist-test-slave-3h5h"
server_key: "c8088b594fbc8dff6b0b350411d9641f"
server_key_iv: "c19e294817e0c71c642c2c66ccfdb1a4"
server_key_version: "encryptionkey@0"
I20260807 02:57:14.116243 16098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:14.131541 16098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:14.135725 16098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:14.136040 16098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:14.137059 16098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:57:14.138530 16098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:57:14.138646 16098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:14.138809 16098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:57:14.138887 16098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:14.174865 16098 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:41833
I20260807 02:57:14.174912 16266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:41833 every 8 connection(s)
I20260807 02:57:14.176282 16098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:57:14.180375 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 16098
I20260807 02:57:14.180631 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:57:14.181366 1509 external_mini_cluster.cc:1428] Setting key e222a1736596a7d541211f2e3bf34e35
Aug 07 02:57:14 dist-test-slave-3h5h krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071434, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:57:14.198009 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:14.178992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:59205 (local address 127.1.121.126:33251)
0807 02:57:14.179469 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:14.179474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:14.180377 (+ 903us) server_negotiation.cc:407] Connection header received
0807 02:57:14.182623 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:14.182628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:14.182724 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:14.182850 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:14.184550 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:14.185360 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:14.187025 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:14.187427 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:14.187573 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:14.191433 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:14.191438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:14.191442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:14.191478 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:14.194596 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:14.195394 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:14.195401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:14.195405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:14.195539 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:14.196050 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:14.196055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:14.196060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:14.196345 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:14.196507 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:14.197376 (+ 869us) server_negotiation.cc:299] Negotiation successful
0807 02:57:14.197619 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":254,"threads_started":1}
I20260807 02:57:14.199015 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:14.179422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33251 (local address 127.1.121.65:59205)
0807 02:57:14.180201 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:14.180240 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:14.182299 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:14.183132 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:14.183144 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:14.183731 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:14.184366 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:14.184384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:14.185550 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:14.185556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:14.186842 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:14.186854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:14.187676 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:14.187684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:14.187935 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:14.188655 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:14.188679 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:14.191191 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:57:14.194761 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:57:14.194773 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:57:14.194792 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:57:14.195256 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:57:14.195669 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:57:14.195674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:57:14.195678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:57:14.195927 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:57:14.196484 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:57:14.196493 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:57:14.196978 (+ 485us) client_negotiation.cc:769] Sending connection context
0807 02:57:14.197269 (+ 291us) client_negotiation.cc:240] Negotiation successful
0807 02:57:14.197841 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":328,"threads_started":1}
I20260807 02:57:14.200706 16267 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33251
I20260807 02:57:14.201082 16267 heartbeater.cc:464] Registering TS with master...
I20260807 02:57:14.202093 16267 heartbeater.cc:511] Master 127.1.121.126:33251 requested a full tablet report, sending...
I20260807 02:57:14.204710 16020 ts_manager.cc:194] Registered new tserver with Master: a69a0b67b8544c1bbf866921cdbe4306 (127.1.121.65:41833)
I20260807 02:57:14.207041 16020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:59205
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:57:14.256242 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33251
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:57:14.622450 16274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:14.622768 16274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:14.622964 16274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:14.630156 16274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:14.630285 16274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:14.630468 16274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:57:14.641108 16274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:14.642469 16274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:14.643813 16274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:14.657685 16281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:14.657931 16285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:14.658175 16282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:14.658824 16274 server_base.cc:1034] running on GCE node
I20260807 02:57:14.659510 16274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:14.660576 16274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:14.661823 16274 hybrid_clock.cc:648] HybridClock initialized: now 1786071434661738 us; error 88 us; skew 500 ppm
I20260807 02:57:14.662246 16274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:57:14 dist-test-slave-3h5h krb5kdc[15960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071434, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:57:14.665912 16274 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:57:14.667678 16274 webserver.cc:460] Webserver started at http://127.1.121.66:43943/ using document root <none> and password file <none>
I20260807 02:57:14.668450 16274 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:14.668615 16274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:14.669034 16274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:14.672374 16274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5"
format_stamp: "Formatted at 2026-08-07 02:57:14 on dist-test-slave-3h5h"
server_key: "a2435d4268903111ecdfdf69fb4ae878"
server_key_iv: "8a4e25bbcb7513b7c2902999494f4c29"
server_key_version: "encryptionkey@0"
I20260807 02:57:14.673360 16274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5"
format_stamp: "Formatted at 2026-08-07 02:57:14 on dist-test-slave-3h5h"
server_key: "a2435d4268903111ecdfdf69fb4ae878"
server_key_iv: "8a4e25bbcb7513b7c2902999494f4c29"
server_key_version: "encryptionkey@0"
I20260807 02:57:14.679631 16274 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 02:57:14.683637 16295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:14.685142 16274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:57:14.685335 16274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5"
format_stamp: "Formatted at 2026-08-07 02:57:14 on dist-test-slave-3h5h"
server_key: "a2435d4268903111ecdfdf69fb4ae878"
server_key_iv: "8a4e25bbcb7513b7c2902999494f4c29"
server_key_version: "encryptionkey@0"
I20260807 02:57:14.685539 16274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:14.712756 16274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:14.717090 16274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:14.717397 16274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:14.718516 16274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:57:14.720011 16274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:57:14.720114 16274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:14.720295 16274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:57:14.720394 16274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 02:57:14.757304 16274 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:32805
I20260807 02:57:14.757349 16447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:32805 every 8 connection(s)
I20260807 02:57:14.758656 16274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:57:14.759044 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 16274
I20260807 02:57:14.759274 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:57:14.759928 1509 external_mini_cluster.cc:1428] Setting key 8869776842ba1b3bc6f5f543d160c252
Aug 07 02:57:14 dist-test-slave-3h5h krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071434, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:57:14.783486 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:14.761679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:37113 (local address 127.1.121.126:33251)
0807 02:57:14.764439 (+ 2760us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:14.764447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:14.764472 (+ 25us) server_negotiation.cc:407] Connection header received
0807 02:57:14.765762 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:14.765770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:14.765864 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:14.765997 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:14.767496 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:14.768712 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:14.771010 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:14.771752 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:14.771917 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:14.776784 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:14.776793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:14.776801 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:14.776873 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:14.779861 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:14.780654 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:14.780664 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:14.780666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:14.781023 (+ 357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:14.781585 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:14.781591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:14.781596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:14.781898 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:14.782091 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:14.782835 (+ 744us) server_negotiation.cc:299] Negotiation successful
0807 02:57:14.783085 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2644,"thread_start_us":2541,"threads_started":1}
I20260807 02:57:14.784552 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:14.762498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33251 (local address 127.1.121.66:37113)
0807 02:57:14.763465 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:14.763512 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:14.765464 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:14.766194 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:14.766202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:14.766745 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:14.767311 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:14.767327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:14.768912 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:14.768920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:14.770831 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:14.770848 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:14.771994 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:14.772002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:14.772311 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:14.773153 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:14.773186 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:14.776449 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:57:14.780025 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:57:14.780035 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:57:14.780054 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:57:14.780507 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:57:14.781174 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:57:14.781180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:57:14.781184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:57:14.781411 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:57:14.782056 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:57:14.782064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:57:14.782434 (+ 370us) client_negotiation.cc:769] Sending connection context
0807 02:57:14.782733 (+ 299us) client_negotiation.cc:240] Negotiation successful
0807 02:57:14.783316 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":399,"threads_started":1}
I20260807 02:57:14.786201 16449 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33251
I20260807 02:57:14.786703 16449 heartbeater.cc:464] Registering TS with master...
I20260807 02:57:14.787766 16449 heartbeater.cc:511] Master 127.1.121.126:33251 requested a full tablet report, sending...
I20260807 02:57:14.789875 16020 ts_manager.cc:194] Registered new tserver with Master: e3f1aa4e12f647bdb7c3b33c61559ee5 (127.1.121.66:32805)
I20260807 02:57:14.791379 16020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:37113
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:57:14.840075 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33251
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:57:15.211133 16267 heartbeater.cc:503] Master 127.1.121.126:33251 was elected leader, sending a full tablet report...
W20260807 02:57:15.228143 16457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:57:15.228420 16457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:57:15.228518 16457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:57:15.236053 16457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:57:15.236172 16457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:57:15.236289 16457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:57:15.246104 16457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:57:15.247382 16457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:57:15.248756 16457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:57:15.263149 16464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:15.263186 16469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:57:15.263243 16465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:57:15.264554 16457 server_base.cc:1034] running on GCE node
I20260807 02:57:15.265316 16457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:57:15.266460 16457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:57:15.267717 16457 hybrid_clock.cc:648] HybridClock initialized: now 1786071435267657 us; error 42 us; skew 500 ppm
I20260807 02:57:15.268074 16457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:57:15 dist-test-slave-3h5h krb5kdc[15960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071435, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:57:15.271549 16457 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:57:15.273223 16457 webserver.cc:460] Webserver started at http://127.1.121.67:46825/ using document root <none> and password file <none>
I20260807 02:57:15.273974 16457 fs_manager.cc:359] Metadata directory not provided
I20260807 02:57:15.274148 16457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:57:15.274473 16457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:57:15.277546 16457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "7ea4bae903b54a6992f892aa430e523e"
format_stamp: "Formatted at 2026-08-07 02:57:15 on dist-test-slave-3h5h"
server_key: "b5737d5b403618c31cc100afeb3c2a02"
server_key_iv: "617bebd25e8d93af12c0d734428cef13"
server_key_version: "encryptionkey@0"
I20260807 02:57:15.278352 16457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "7ea4bae903b54a6992f892aa430e523e"
format_stamp: "Formatted at 2026-08-07 02:57:15 on dist-test-slave-3h5h"
server_key: "b5737d5b403618c31cc100afeb3c2a02"
server_key_iv: "617bebd25e8d93af12c0d734428cef13"
server_key_version: "encryptionkey@0"
I20260807 02:57:15.283629 16457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 02:57:15.287784 16479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:15.289268 16457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 02:57:15.289451 16457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "7ea4bae903b54a6992f892aa430e523e"
format_stamp: "Formatted at 2026-08-07 02:57:15 on dist-test-slave-3h5h"
server_key: "b5737d5b403618c31cc100afeb3c2a02"
server_key_iv: "617bebd25e8d93af12c0d734428cef13"
server_key_version: "encryptionkey@0"
I20260807 02:57:15.289661 16457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:57:15.310616 16457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:57:15.315685 16457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:57:15.316002 16457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:57:15.317102 16457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:57:15.318655 16457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:57:15.318769 16457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:15.318922 16457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:57:15.318996 16457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:57:15.354303 16457 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:44099
I20260807 02:57:15.354417 16629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:44099 every 8 connection(s)
I20260807 02:57:15.355656 16457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:57:15.356196 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 16457
I20260807 02:57:15.356420 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:57:15.357224 1509 external_mini_cluster.cc:1428] Setting key 9f5957716a1c32e936eb2a85c1160028
Aug 07 02:57:15 dist-test-slave-3h5h krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071435, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:57:15.377830 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.358846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:42881 (local address 127.1.121.126:33251)
0807 02:57:15.359383 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.359388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.360770 (+ 1382us) server_negotiation.cc:407] Connection header received
0807 02:57:15.362757 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.362762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.362834 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.362933 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:15.364454 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.365456 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.367055 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.367555 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.367668 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.371515 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:15.371521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:15.371525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:15.371563 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:15.374726 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:15.375536 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:15.375541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:15.375543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:15.375674 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:15.376126 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:15.376129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:15.376131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:15.376405 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:15.376525 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.377367 (+ 842us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.377545 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":316,"threads_started":1}
I20260807 02:57:15.378923 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.359535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33251 (local address 127.1.121.67:42881)
0807 02:57:15.360563 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:15.360651 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:15.362495 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:15.363194 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:15.363202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:15.363710 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:57:15.364293 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.364311 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.365616 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.365620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.366906 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.366918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.367766 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.367771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.367976 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.368704 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:57:15.368728 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:57:15.371229 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:57:15.374867 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:57:15.374875 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:57:15.374891 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:57:15.375399 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:57:15.375783 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:57:15.375787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:57:15.375789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:57:15.376007 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:57:15.376546 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:57:15.376552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:57:15.376987 (+ 435us) client_negotiation.cc:769] Sending connection context
0807 02:57:15.377259 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 02:57:15.377690 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":362,"threads_started":1}
I20260807 02:57:15.380604 16631 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33251
I20260807 02:57:15.380995 16631 heartbeater.cc:464] Registering TS with master...
I20260807 02:57:15.381786 16631 heartbeater.cc:511] Master 127.1.121.126:33251 requested a full tablet report, sending...
I20260807 02:57:15.383319 16020 ts_manager.cc:194] Registered new tserver with Master: 7ea4bae903b54a6992f892aa430e523e (127.1.121.67:44099)
I20260807 02:57:15.384194 16020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:42881
I20260807 02:57:15.386428 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:57:15.405159 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.391503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.1.121.126:33251)
0807 02:57:15.391675 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.391679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.392302 (+ 623us) server_negotiation.cc:407] Connection header received
0807 02:57:15.392518 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.392529 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.392590 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.392693 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:15.394300 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.395022 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.396315 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.396725 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.396891 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.399255 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:15.399260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:15.399263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:15.399293 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:15.401970 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:15.402836 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:15.402844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:15.402848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:15.403012 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:15.403443 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:15.403449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:15.403453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:15.403825 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:15.404126 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.404519 (+ 393us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.404800 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:57:15.409679 16020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:57:15.416706 16020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:57:15.440773 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.429732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.1.121.66:32805)
0807 02:57:15.430337 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.430344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.431360 (+ 1016us) server_negotiation.cc:407] Connection header received
0807 02:57:15.431769 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.431778 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.432037 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.432209 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:57:15.433815 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.435528 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.438022 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.439507 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.439660 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.439873 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.440251 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.440423 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":352,"threads_started":1}
I20260807 02:57:15.440896 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.429672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:32805 (local address 127.0.0.1:34484)
0807 02:57:15.431204 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:15.431242 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:15.431451 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:15.433094 (+ 1643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:15.433105 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:15.433147 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:57:15.433580 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.433596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.435697 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.435705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.437852 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.437865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.439722 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.439729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.439942 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.440001 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 02:57:15.440201 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 02:57:15.440413 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"mutex_wait_us":2,"thread_start_us":1854,"threads_started":1}
I20260807 02:57:15.441026 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.429722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:41833 (local address 127.0.0.1:56304)
0807 02:57:15.431805 (+ 2083us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:15.431822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:15.431925 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:15.433615 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:15.433625 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:15.433653 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:57:15.434070 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.434082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.435996 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.436004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.438096 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.438111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.440041 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.440048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.440237 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.440270 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 02:57:15.440363 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 02:57:15.440512 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1961,"mutex_wait_us":57,"thread_start_us":1200,"threads_started":1}
I20260807 02:57:15.441960 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.430149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.1.121.65:41833)
0807 02:57:15.433011 (+ 2862us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.433019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.433047 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:57:15.433169 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.433179 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.433409 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.433612 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:57:15.434307 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.435862 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.438306 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.439743 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.439892 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.440141 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.441434 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.441607 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2688,"thread_start_us":2530,"threads_started":1}
I20260807 02:57:15.442196 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.431918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:44099 (local address 127.0.0.1:55000)
0807 02:57:15.432681 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:15.432698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:15.432813 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:15.434213 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:15.434217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:15.434241 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:57:15.434950 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.434965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.437133 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.437139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.439316 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.439332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.441466 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.441474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.441653 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.441684 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 02:57:15.441781 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 02:57:15.441898 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":518,"threads_started":1}
I20260807 02:57:15.443487 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.432609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.1.121.67:44099)
0807 02:57:15.433629 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.433637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.433670 (+ 33us) server_negotiation.cc:407] Connection header received
0807 02:57:15.433802 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.433810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.434019 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.434213 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:57:15.435186 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.436969 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.439493 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.440953 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.441402 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.441633 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.441837 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.442010 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":614,"threads_started":1}
I20260807 02:57:15.445489 16365 tablet_service.cc:1467] Processing CreateTablet for tablet 309530d888be4e58b2afe67288787ce9 (DEFAULT_TABLE table=test-table [id=318986af4a6a4a60a6ae25c62c3d60b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:57:15.445756 16178 tablet_service.cc:1467] Processing CreateTablet for tablet 309530d888be4e58b2afe67288787ce9 (DEFAULT_TABLE table=test-table [id=318986af4a6a4a60a6ae25c62c3d60b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:57:15.446903 16365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309530d888be4e58b2afe67288787ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:57:15.447140 16178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309530d888be4e58b2afe67288787ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:57:15.447336 16537 tablet_service.cc:1467] Processing CreateTablet for tablet 309530d888be4e58b2afe67288787ce9 (DEFAULT_TABLE table=test-table [id=318986af4a6a4a60a6ae25c62c3d60b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:57:15.449107 16537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 309530d888be4e58b2afe67288787ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:57:15.460026 16652 tablet_bootstrap.cc:492] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: Bootstrap starting.
I20260807 02:57:15.461881 16654 tablet_bootstrap.cc:492] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: Bootstrap starting.
I20260807 02:57:15.464438 16655 tablet_bootstrap.cc:492] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: Bootstrap starting.
I20260807 02:57:15.464666 16652 tablet_bootstrap.cc:654] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: Neither blocks nor log segments found. Creating new log.
I20260807 02:57:15.465684 16654 tablet_bootstrap.cc:654] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: Neither blocks nor log segments found. Creating new log.
I20260807 02:57:15.466068 16652 log.cc:824] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: Log is configured to *not* fsync() on all Append() calls
I20260807 02:57:15.466853 16654 log.cc:824] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: Log is configured to *not* fsync() on all Append() calls
I20260807 02:57:15.468006 16655 tablet_bootstrap.cc:654] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: Neither blocks nor log segments found. Creating new log.
I20260807 02:57:15.469264 16655 log.cc:824] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:57:15.469451 16652 tablet_bootstrap.cc:492] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: No bootstrap required, opened a new log
I20260807 02:57:15.469743 16652 ts_tablet_manager.cc:1397] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 02:57:15.470676 16654 tablet_bootstrap.cc:492] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: No bootstrap required, opened a new log
I20260807 02:57:15.470947 16654 ts_tablet_manager.cc:1397] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 02:57:15.472493 16655 tablet_bootstrap.cc:492] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: No bootstrap required, opened a new log
I20260807 02:57:15.472754 16655 ts_tablet_manager.cc:1397] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260807 02:57:15.478471 16654 raft_consensus.cc:348] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.478873 16654 raft_consensus.cc:374] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:57:15.478976 16654 raft_consensus.cc:729] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69a0b67b8544c1bbf866921cdbe4306, State: Initialized, Role: FOLLOWER
I20260807 02:57:15.479619 16654 consensus_queue.cc:260] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.480523 16652 raft_consensus.cc:348] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.480978 16652 raft_consensus.cc:374] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:57:15.481079 16652 raft_consensus.cc:729] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f1aa4e12f647bdb7c3b33c61559ee5, State: Initialized, Role: FOLLOWER
I20260807 02:57:15.481657 16652 consensus_queue.cc:260] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.483810 16655 raft_consensus.cc:348] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.484417 16655 raft_consensus.cc:374] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:57:15.484591 16655 raft_consensus.cc:729] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ea4bae903b54a6992f892aa430e523e, State: Initialized, Role: FOLLOWER
I20260807 02:57:15.485607 16654 ts_tablet_manager.cc:1428] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: Time spent starting tablet: real 0.015s user 0.003s sys 0.009s
I20260807 02:57:15.485349 16655 consensus_queue.cc:260] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.486545 16449 heartbeater.cc:503] Master 127.1.121.126:33251 was elected leader, sending a full tablet report...
I20260807 02:57:15.486976 16631 heartbeater.cc:503] Master 127.1.121.126:33251 was elected leader, sending a full tablet report...
I20260807 02:57:15.487231 16652 ts_tablet_manager.cc:1428] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
I20260807 02:57:15.488584 16655 ts_tablet_manager.cc:1428] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260807 02:57:15.511215 16451 tablet.cc:2369] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:57:15.607755 16632 tablet.cc:2369] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:57:15.679220 16268 tablet.cc:2369] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:57:15.730767 16661 raft_consensus.cc:482] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:57:15.731091 16661 raft_consensus.cc:504] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.732826 16661 leader_election.cc:288] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a69a0b67b8544c1bbf866921cdbe4306 (127.1.121.65:41833), 7ea4bae903b54a6992f892aa430e523e (127.1.121.67:44099)
I20260807 02:57:15.740049 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.733456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:45611 (local address 127.1.121.65:41833)
0807 02:57:15.733742 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.733751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.734656 (+ 905us) server_negotiation.cc:407] Connection header received
0807 02:57:15.734899 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.734903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.734965 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.735064 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:57:15.735644 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.736379 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.738279 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.739084 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.739281 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.739360 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.739655 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.739782 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 02:57:15.740206 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.733498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:33203 (local address 127.1.121.67:44099)
0807 02:57:15.733768 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.733776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.734853 (+ 1077us) server_negotiation.cc:407] Connection header received
0807 02:57:15.735052 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.735058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.735127 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.735229 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:57:15.735879 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.736615 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.738420 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.739199 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.739410 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.739496 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.739799 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.739928 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 02:57:15.740275 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.733320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:41833 (local address 127.1.121.66:45611)
0807 02:57:15.734524 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:15.734543 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:15.734736 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:15.735083 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:15.735088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:15.735113 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:57:15.735482 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.735491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.736510 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.736518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.738114 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.738126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.739316 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.739322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.739477 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.739500 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:57:15.739629 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 02:57:15.739756 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"mutex_wait_us":2,"thread_start_us":848,"threads_started":1}
I20260807 02:57:15.740381 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.733540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:44099 (local address 127.1.121.66:33203)
0807 02:57:15.734732 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:15.734747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:15.734904 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:15.735252 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:15.735261 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:15.735312 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:57:15.735718 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.735726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.736766 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.736773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.738250 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.738263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.739437 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.739443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.739638 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.739661 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:57:15.739770 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 02:57:15.739895 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":3,"spinlock_wait_cycles":1792,"thread_start_us":684,"threads_started":1}
I20260807 02:57:15.741690 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309530d888be4e58b2afe67288787ce9" candidate_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ea4bae903b54a6992f892aa430e523e" is_pre_election: true
I20260807 02:57:15.741690 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309530d888be4e58b2afe67288787ce9" candidate_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69a0b67b8544c1bbf866921cdbe4306" is_pre_election: true
I20260807 02:57:15.742323 16202 raft_consensus.cc:2454] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f1aa4e12f647bdb7c3b33c61559ee5 in term 0.
I20260807 02:57:15.742343 16562 raft_consensus.cc:2454] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f1aa4e12f647bdb7c3b33c61559ee5 in term 0.
I20260807 02:57:15.742581 16662 raft_consensus.cc:482] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:57:15.742825 16662 raft_consensus.cc:504] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.743148 16296 leader_election.cc:302] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7ea4bae903b54a6992f892aa430e523e, a69a0b67b8544c1bbf866921cdbe4306, e3f1aa4e12f647bdb7c3b33c61559ee5; no voters:
I20260807 02:57:15.743579 16661 raft_consensus.cc:2781] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:57:15.743736 16661 raft_consensus.cc:482] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:57:15.743878 16661 raft_consensus.cc:3037] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:57:15.744671 16662 leader_election.cc:288] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3f1aa4e12f647bdb7c3b33c61559ee5 (127.1.121.66:32805), a69a0b67b8544c1bbf866921cdbe4306 (127.1.121.65:41833)
I20260807 02:57:15.747404 16661 raft_consensus.cc:504] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.748085 16661 leader_election.cc:288] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [CANDIDATE]: Term 1 election: Requested vote from peers a69a0b67b8544c1bbf866921cdbe4306 (127.1.121.65:41833), 7ea4bae903b54a6992f892aa430e523e (127.1.121.67:44099)
I20260807 02:57:15.748606 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309530d888be4e58b2afe67288787ce9" candidate_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69a0b67b8544c1bbf866921cdbe4306"
I20260807 02:57:15.748862 16202 raft_consensus.cc:3037] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:57:15.749109 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309530d888be4e58b2afe67288787ce9" candidate_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ea4bae903b54a6992f892aa430e523e"
I20260807 02:57:15.749336 16562 raft_consensus.cc:3037] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:57:15.752308 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.745011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:32805 (local address 127.1.121.67:40669)
0807 02:57:15.745206 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:15.745239 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:15.745356 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:15.745820 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:15.745826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:15.745898 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:57:15.746115 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.746128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.747494 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.747500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.749715 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.749730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.751496 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.751502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.751718 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.751758 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 02:57:15.751852 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 02:57:15.752002 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 02:57:15.752315 16202 raft_consensus.cc:2454] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f1aa4e12f647bdb7c3b33c61559ee5 in term 1.
I20260807 02:57:15.752790 16299 leader_election.cc:302] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69a0b67b8544c1bbf866921cdbe4306, e3f1aa4e12f647bdb7c3b33c61559ee5; no voters:
I20260807 02:57:15.753153 16661 raft_consensus.cc:2781] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:57:15.753321 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.746142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:52141 (local address 127.1.121.65:41833)
0807 02:57:15.746325 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.746333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.746351 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:57:15.746407 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.746412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.746486 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.746586 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:57:15.747109 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.748141 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.750736 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.752119 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.752268 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.752358 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.752923 (+ 565us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.753052 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:57:15.753378 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.745351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:41833 (local address 127.1.121.67:52141)
0807 02:57:15.746150 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0807 02:57:15.746168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:57:15.746284 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:57:15.746603 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:57:15.746607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:57:15.746632 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:57:15.746959 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.746967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.748328 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.748334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.750527 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:57:15.750546 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.752561 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.752567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:57:15.752724 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.752754 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 02:57:15.752893 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 02:57:15.753027 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":373,"threads_started":1}
I20260807 02:57:15.753736 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.745178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:40669 (local address 127.1.121.66:32805)
0807 02:57:15.745357 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.745363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.745384 (+ 21us) server_negotiation.cc:407] Connection header received
0807 02:57:15.745454 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.745459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.745535 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.745628 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:57:15.746303 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.747338 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.749901 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.751193 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.751339 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.751407 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.752587 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.752717 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:57:15.754120 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309530d888be4e58b2afe67288787ce9" candidate_uuid: "7ea4bae903b54a6992f892aa430e523e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69a0b67b8544c1bbf866921cdbe4306" is_pre_election: true
I20260807 02:57:15.754503 16202 raft_consensus.cc:2379] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ea4bae903b54a6992f892aa430e523e in current term 1: Already voted for candidate e3f1aa4e12f647bdb7c3b33c61559ee5 in this term.
I20260807 02:57:15.754503 16562 raft_consensus.cc:2454] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f1aa4e12f647bdb7c3b33c61559ee5 in term 1.
I20260807 02:57:15.754758 16391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "309530d888be4e58b2afe67288787ce9" candidate_uuid: "7ea4bae903b54a6992f892aa430e523e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" is_pre_election: true
I20260807 02:57:15.755298 16661 raft_consensus.cc:686] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 1 LEADER]: Becoming Leader. State: Replica: e3f1aa4e12f647bdb7c3b33c61559ee5, State: Running, Role: LEADER
I20260807 02:57:15.755823 16661 consensus_queue.cc:237] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } }
I20260807 02:57:15.760349 16480 leader_election.cc:302] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ea4bae903b54a6992f892aa430e523e; no voters: a69a0b67b8544c1bbf866921cdbe4306, e3f1aa4e12f647bdb7c3b33c61559ee5
I20260807 02:57:15.760877 16662 raft_consensus.cc:2726] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 02:57:15.761911 16019 catalog_manager.cc:5495] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3f1aa4e12f647bdb7c3b33c61559ee5 (127.1.121.66). New cstate: current_term: 1 leader_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f1aa4e12f647bdb7c3b33c61559ee5" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 32805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 } health_report { overall_health: UNKNOWN } } }
I20260807 02:57:15.814746 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.809179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.1.121.66:32805)
0807 02:57:15.809405 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.809410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.809428 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:57:15.809637 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.809641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.809739 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.809866 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:57:15.810319 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.811061 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.812522 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.812942 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.813031 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.813559 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:57:15.814093 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:57:15.814206 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.814338 (+ 132us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.814402 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 02:57:15.823268 16202 raft_consensus.cc:1261] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 1 FOLLOWER]: Refusing update from remote peer e3f1aa4e12f647bdb7c3b33c61559ee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:57:15.823388 16562 raft_consensus.cc:1261] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 1 FOLLOWER]: Refusing update from remote peer e3f1aa4e12f647bdb7c3b33c61559ee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:57:15.824294 16667 consensus_queue.cc:1035] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ea4bae903b54a6992f892aa430e523e" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:57:15.824659 16661 consensus_queue.cc:1035] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69a0b67b8544c1bbf866921cdbe4306" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 41833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:57:15.856178 16019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:57:15.856603 16019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:57:15.861032 16019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3106995c61cd42b285375b83e240bf32: Sending DeleteTablet for 3 replicas of tablet 309530d888be4e58b2afe67288787ce9
I20260807 02:57:15.862346 16537 tablet_service.cc:1514] Processing DeleteTablet for tablet 309530d888be4e58b2afe67288787ce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:57:15 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:55000
I20260807 02:57:15.862356 16178 tablet_service.cc:1514] Processing DeleteTablet for tablet 309530d888be4e58b2afe67288787ce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:57:15 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:56304
I20260807 02:57:15.862581 16365 tablet_service.cc:1514] Processing DeleteTablet for tablet 309530d888be4e58b2afe67288787ce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:57:15 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:34484
I20260807 02:57:15.864588 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 16098
I20260807 02:57:15.877652 16686 tablet_replica.cc:330] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: stopping tablet replica
I20260807 02:57:15.878187 16686 raft_consensus.cc:2229] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:57:15.878839 16686 raft_consensus.cc:2258] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:57:15.881335 16686 ts_tablet_manager.cc:1905] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:57:15.884325 16685 tablet_replica.cc:330] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: stopping tablet replica
I20260807 02:57:15.884999 16690 tablet_replica.cc:330] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: stopping tablet replica
I20260807 02:57:15.885543 16685 raft_consensus.cc:2229] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:57:15.886078 16690 raft_consensus.cc:2229] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:57:15.886699 16685 raft_consensus.cc:2258] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:57:15.887501 16690 raft_consensus.cc:2258] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:57:15.888789 16686 ts_tablet_manager.cc:1918] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:57:15.889078 16686 log.cc:1196] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/309530d888be4e58b2afe67288787ce9
I20260807 02:57:15.889680 16686 ts_tablet_manager.cc:1939] T 309530d888be4e58b2afe67288787ce9 P a69a0b67b8544c1bbf866921cdbe4306: Deleting consensus metadata
Aug 07 02:57:15 dist-test-slave-3h5h krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:57:15.892055 16003 catalog_manager.cc:4841] TS a69a0b67b8544c1bbf866921cdbe4306 (127.1.121.65:41833): tablet 309530d888be4e58b2afe67288787ce9 (table test-table [id=318986af4a6a4a60a6ae25c62c3d60b2]) successfully deleted
I20260807 02:57:15.894802 16685 ts_tablet_manager.cc:1905] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:57:15.895272 16690 ts_tablet_manager.cc:1905] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:57:15.901834 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:15.866483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.1.121.65:41833)
0807 02:57:15.866719 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:15.866729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:15.878281 (+ 11552us) server_negotiation.cc:407] Connection header received
0807 02:57:15.878760 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:15.878766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:15.878866 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:15.878949 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:15.880947 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.881990 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.884927 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:15.885616 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:15.885726 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:15.893082 (+ 7356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:15.893099 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:15.893119 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:15.893169 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:15.897280 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:15.898300 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:15.898310 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:15.898314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:15.898481 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:15.898933 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:15.898937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:15.898941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:15.899452 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:15.899825 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:15.901085 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0807 02:57:15.901386 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 02:57:15.902516 16685 ts_tablet_manager.cc:1918] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:57:15.902689 16685 log.cc:1196] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/309530d888be4e58b2afe67288787ce9
I20260807 02:57:15.903103 16261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:15.904011 16690 ts_tablet_manager.cc:1918] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:57:15.904237 16690 log.cc:1196] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/309530d888be4e58b2afe67288787ce9
I20260807 02:57:15.905222 16690 ts_tablet_manager.cc:1939] T 309530d888be4e58b2afe67288787ce9 P e3f1aa4e12f647bdb7c3b33c61559ee5: Deleting consensus metadata
I20260807 02:57:15.907985 15999 catalog_manager.cc:4841] TS e3f1aa4e12f647bdb7c3b33c61559ee5 (127.1.121.66:32805): tablet 309530d888be4e58b2afe67288787ce9 (table test-table [id=318986af4a6a4a60a6ae25c62c3d60b2]) successfully deleted
I20260807 02:57:15.908085 16685 ts_tablet_manager.cc:1939] T 309530d888be4e58b2afe67288787ce9 P 7ea4bae903b54a6992f892aa430e523e: Deleting consensus metadata
I20260807 02:57:15.909873 15999 catalog_manager.cc:4841] TS 7ea4bae903b54a6992f892aa430e523e (127.1.121.67:44099): tablet 309530d888be4e58b2afe67288787ce9 (table test-table [id=318986af4a6a4a60a6ae25c62c3d60b2]) successfully deleted
I20260807 02:57:15.997896 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 16098
I20260807 02:57:16.032811 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 16274
Aug 07 02:57:16 dist-test-slave-3h5h krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:57:16.051232 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:16.033977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.1.121.66:32805)
0807 02:57:16.034214 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:16.034220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:16.034244 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:57:16.034323 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:16.034330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:16.034412 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:16.034548 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:16.035820 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:16.037047 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:16.038291 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:16.038958 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:16.039093 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:16.043629 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:16.043651 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:16.043681 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:16.043727 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:16.047968 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:16.048648 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:16.048659 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:16.048663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:16.048890 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:16.049330 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:16.049335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:16.049339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:16.049868 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:16.050184 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:16.050610 (+ 426us) server_negotiation.cc:299] Negotiation successful
0807 02:57:16.050854 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 02:57:16.052170 16443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:16.169979 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 16274
I20260807 02:57:16.202231 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 16457
Aug 07 02:57:16 dist-test-slave-3h5h krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:57:16.218423 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:57:16.203068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.1.121.67:44099)
0807 02:57:16.203321 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 02:57:16.203326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:57:16.203350 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:57:16.203424 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:57:16.203430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:57:16.203552 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:57:16.203657 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:57:16.205003 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:16.205748 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:16.207003 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:57:16.207393 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:57:16.207487 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:57:16.212183 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:57:16.212200 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:57:16.212218 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:57:16.212252 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:57:16.215481 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:16.216187 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:16.216194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:16.216197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:16.216333 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:57:16.216794 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:57:16.216797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:57:16.216799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:57:16.217257 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:57:16.217482 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 02:57:16.217832 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 02:57:16.217995 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260807 02:57:16.219100 16623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:16.311745 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 16457
I20260807 02:57:16.335630 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 15979
I20260807 02:57:16.336931 16061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:57:16.448473 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 15979
2026-08-07T02:57:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5132 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:57:16 dist-test-slave-3h5h krb5kdc[16700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:57:16 dist-test-slave-3h5h krb5kdc[16700](info): set up 2 sockets
Aug 07 02:57:16 dist-test-slave-3h5h krb5kdc[16700](info): commencing operation
krb5kdc: starting...
W20260807 02:57:17.829470 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.010s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:57:18 dist-test-slave-3h5h krb5kdc[16700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:57:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.121.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.121.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/rangeradmin_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/rangeradmin_127.1.121.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.121.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.121.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/rangerlookup_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/rangerlookup_127.1.121.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.121.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.121.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/HTTP_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/HTTP_127.1.121.84@KRBTEST.COM.keytab.
I20260807 02:57:18.293164 1509 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taska5bS7K/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/postgres -l logfile start
2026-08-07 02:57:21.128 UTC [16739] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 02:57:21.128 UTC [16739] LOG: listening on IPv4 address "127.1.121.84", port 46157
2026-08-07 02:57:21.132 UTC [16739] LOG: listening on Unix socket "/tmp/.s.PGSQL.46157"
2026-08-07 02:57:21.159 UTC [16741] LOG: database system was shut down at 2026-08-07 02:57:19 UTC
2026-08-07 02:57:21.163 UTC [16739] LOG: database system is ready to accept connections
I20260807 02:57:22.331349 1509 mini_postgres.cc:96] Postgres bound to 46157
2026-08-07 02:57:22.358 UTC [16752] FATAL: database "slave" does not exist
127.1.121.84:46157 - accepting connections
I20260807 02:57:22.360138 1509 mini_ranger.cc:161] Starting Ranger...
I20260807 02:57:22.397418 1509 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 02:57:22.969038 1509 mini_ranger.cc:87] Created ranger Postgres database
I20260807 02:57:22.969213 1509 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-admin
2026-08-07 02:57:23,353 [I] DB FLAVOR :POSTGRES
2026-08-07 02:57:23,354 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 02:57:23,354 [I] Checking connection..
2026-08-07 02:57:23,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 02:57:23,652 [I] Checking connection passed.
2026-08-07 02:57:23,652 [I] --------- Verifying version history table ---------
2026-08-07 02:57:23,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:57:23,933 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 02:57:23,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:57:24,207 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 02:57:24,208 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 02:57:24,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 02:57:24.453 UTC [16840] WARNING: there is no transaction in progress
2026-08-07 02:57:24,461 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 02:57:24,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:57:24,719 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 02:57:24,719 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 02:57:24,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 02:57:24,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 02:57:25,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3h5h.c.gcp-upstream.internal','N') ;"
2026-08-07 02:57:25,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 02:57:25,816 [I] Table x_portal_user does not exist in database ranger
2026-08-07 02:57:25,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 02:57:26,079 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 02:57:26,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 02:57:26,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 02:57:26,618 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 02:57:26,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 02:57:27.672 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.690 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.710 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.733 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.756 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.779 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.846 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.854 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.869 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.879 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.889 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.904 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.914 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.920 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.927 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:27.935 UTC [16972] WARNING: there is no transaction in progress
2026-08-07 02:57:28,691 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 02:57:28,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 02:57:28,967 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 02:57:28,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 02:57:29,245 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 02:57:29,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 02:57:29,514 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 02:57:29,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 02:57:29,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 02:57:30,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3h5h.c.gcp-upstream.internal';"
2026-08-07 02:57:30,297 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 02:57:30,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 02:57:30,569 [I] DB_PATCHES have already been applied
I20260807 02:57:30.576292 1509 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-admin:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taska5bS7K/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 02:57:30.576388 1509 mini_ranger.cc:193] Using host: 127.1.121.84
I20260807 02:57:30.600929 1509 mini_ranger.cc:227] Ranger admin URL: http://127.1.121.84:54577
Aug 07, 2026 2:57:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 2:57:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 2:57:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 2:57:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 2:57:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 2:57:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 2:57:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 02:57:32.118633 1509 mini_ranger.cc:160] Time spent starting Ranger: real 9.759s user 0.002s sys 0.075s
Aug 07, 2026 2:57:53 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 2:57:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 2:57:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 2:57:53 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 2:57:54 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 02:57:55.055595 1509 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.121.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.121.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/rangerkms_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/rangerkms_127.1.121.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.121.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.121.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.121.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/HTTP_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.121.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/HTTP_127.1.121.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 02:57:55 dist-test-slave-3h5h krb5kdc[16700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071475, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 02:57:55.335798 1509 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 02:57:55.367389 1509 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 02:57:56.350000 1509 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 02:57:56.350221 1509 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-kms
I20260807 02:57:56.350368 1509 mini_ranger_kms.cc:227] Using postgres at 127.1.121.84:46157
2026-08-07 02:57:57,350 [I] DB FLAVOR :POSTGRES
2026-08-07 02:57:57,350 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 02:57:57,350 [I] Checking connection
2026-08-07 02:57:57,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 02:57:57,644 [I] connection success
2026-08-07 02:57:57,644 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 02:57:57,645 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 02:57:57,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 02:57:57,916 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 02:57:57,916 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 02:57:57,916 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 02:57:57,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:46157/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 02:57:58,237 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 02:57:58.770983 1509 mini_ranger_kms.cc:310] Created kms service
I20260807 02:57:58.984383 1509 mini_ranger_kms.cc:326] Created kudu user
I20260807 02:57:59.035429 1509 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 02:57:59 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071475, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:57:59.438961 1509 mini_ranger_kms.cc:384] Added ranger policy
I20260807 02:57:59.440544 1509 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-kms:/tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 02:57:59.440634 1509 mini_ranger_kms.cc:240] Using host: 127.1.121.84
I20260807 02:57:59.468192 1509 mini_ranger_kms.cc:276] Ranger KMS PID: 17216
I20260807 02:57:59.468410 1509 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.121.84:45715
Aug 07, 2026 2:58:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 2:58:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 2:58:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 2:58:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 2:58:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 2:58:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 02:58:01 dist-test-slave-3h5h krb5kdc[16700](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786071481, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.121.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:58:02.315521 1509 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.980s user 0.136s sys 0.473s
I20260807 02:58:02.315866 1509 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 02:58:02.316107 1509 mini_ranger_kms.cc:401] 127.1.121.84:45715/kms/v1/keys
Aug 07 02:58:02 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071481, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.121.84@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
Aug 07 02:58:03 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071481, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.121.84@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
Aug 07 02:58:05 dist-test-slave-3h5h krb5kdc[16700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:58:05.618366 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:40765
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:40765
--ranger_config_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:45715/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:58:06.023957 17282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:58:06.024413 17282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:58:06.024595 17282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:58:06.033521 17282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:58:06.033738 17282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:58:06.033881 17282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:58:06.034006 17282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:58:06.034117 17282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:58:06.045717 17282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.121.84:45715/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:40765
--ranger_config_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:40765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:58:06.047231 17282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:58:06.048633 17282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:58:06.062723 17287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:58:06.062860 17290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:58:06.062758 17288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:58:06.064076 17282 server_base.cc:1034] running on GCE node
I20260807 02:58:06.064993 17282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:58:06.066548 17282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:58:06.067798 17282 hybrid_clock.cc:648] HybridClock initialized: now 1786071486067761 us; error 38 us; skew 500 ppm
I20260807 02:58:06.068181 17282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:58:06 dist-test-slave-3h5h krb5kdc[16700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071486, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:58:06.078078 17282 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:58:06.079824 17282 webserver.cc:460] Webserver started at http://127.1.121.126:33155/ using document root <none> and password file <none>
I20260807 02:58:06.080581 17282 fs_manager.cc:359] Metadata directory not provided
I20260807 02:58:06.080749 17282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:58:06.081121 17282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:58:06 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071486, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:58:06.145740 17282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "c52f59be554643da84798d2ee65d1fb0"
format_stamp: "Formatted at 2026-08-07 02:58:06 on dist-test-slave-3h5h"
server_key: "71b74f3b21c4a918e789f9f6f626f477"
server_key_iv: "3f81ade2db190654600002cec3ba8a0a"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:06.146755 17282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "c52f59be554643da84798d2ee65d1fb0"
format_stamp: "Formatted at 2026-08-07 02:58:06 on dist-test-slave-3h5h"
server_key: "71b74f3b21c4a918e789f9f6f626f477"
server_key_iv: "3f81ade2db190654600002cec3ba8a0a"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:06.152976 17282 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260807 02:58:06.173637 17301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:06.175284 17282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:58:06.175490 17282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "c52f59be554643da84798d2ee65d1fb0"
format_stamp: "Formatted at 2026-08-07 02:58:06 on dist-test-slave-3h5h"
server_key: "71b74f3b21c4a918e789f9f6f626f477"
server_key_iv: "3f81ade2db190654600002cec3ba8a0a"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:06.175751 17282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:58:06.198088 17282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:58:06.202872 17282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:58:06.203251 17282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:58:06.226308 17282 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:40765
I20260807 02:58:06.226323 17371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:40765 every 8 connection(s)
I20260807 02:58:06.227783 17282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:58:06.230429 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 17282
I20260807 02:58:06.230659 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:58:06.231292 1509 external_mini_cluster.cc:1428] Setting key 5b9d65110bee8332cda3d3dcdc0cde5d
I20260807 02:58:06.232035 17372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 02:58:06 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:58:06.247752 17372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0: Bootstrap starting.
I20260807 02:58:06.252084 17372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0: Neither blocks nor log segments found. Creating new log.
I20260807 02:58:06.253556 17372 log.cc:824] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0: Log is configured to *not* fsync() on all Append() calls
I20260807 02:58:06.257328 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:06.233418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.1.121.126:40765)
0807 02:58:06.234315 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:06.234331 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:06.234381 (+ 50us) server_negotiation.cc:407] Connection header received
0807 02:58:06.237194 (+ 2813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:06.237234 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:06.237772 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:06.238234 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:58:06.239901 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:06.242022 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:06.243340 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:06.244462 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:06.244675 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:06.248881 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:58:06.248894 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:58:06.248913 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:58:06.248991 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:58:06.253099 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:06.253826 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:06.253834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:06.253841 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:06.254004 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:06.254406 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:06.254415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:06.254423 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:06.254959 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:58:06.255301 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:06.255794 (+ 493us) server_negotiation.cc:299] Negotiation successful
0807 02:58:06.256203 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":383,"threads_started":1}
I20260807 02:58:06.257668 17372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0: No bootstrap required, opened a new log
I20260807 02:58:06.266299 17372 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52f59be554643da84798d2ee65d1fb0" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40765 } }
I20260807 02:58:06.266702 17372 raft_consensus.cc:374] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:58:06.266856 17372 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52f59be554643da84798d2ee65d1fb0, State: Initialized, Role: FOLLOWER
I20260807 02:58:06.267511 17372 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52f59be554643da84798d2ee65d1fb0" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40765 } }
I20260807 02:58:06.267761 17372 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:58:06.267969 17372 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:58:06.268178 17372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:58:06.270754 17372 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52f59be554643da84798d2ee65d1fb0" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40765 } }
I20260807 02:58:06.271308 17372 leader_election.cc:302] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52f59be554643da84798d2ee65d1fb0; no voters:
I20260807 02:58:06.272075 17372 leader_election.cc:288] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:58:06.272480 17380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:58:06.273792 17380 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [term 1 LEADER]: Becoming Leader. State: Replica: c52f59be554643da84798d2ee65d1fb0, State: Running, Role: LEADER
I20260807 02:58:06.274339 17380 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52f59be554643da84798d2ee65d1fb0" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40765 } }
I20260807 02:58:06.274829 17372 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:58:06.278560 17382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52f59be554643da84798d2ee65d1fb0. Latest consensus state: current_term: 1 leader_uuid: "c52f59be554643da84798d2ee65d1fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52f59be554643da84798d2ee65d1fb0" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40765 } } }
I20260807 02:58:06.278923 17382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [sys.catalog]: This master's current role is: LEADER
I20260807 02:58:06.281075 17381 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c52f59be554643da84798d2ee65d1fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52f59be554643da84798d2ee65d1fb0" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40765 } } }
I20260807 02:58:06.281457 17381 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0 [sys.catalog]: This master's current role is: LEADER
I20260807 02:58:06.281700 17389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:58:06.289400 17389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:58:06.290421 17372 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 02:58:06.307914 17389 catalog_manager.cc:1347] Generated new cluster ID: 424d4b553ef141189fc63177bb234020
I20260807 02:58:06.308091 17389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:58:06.359812 17389 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:58:06.362032 17389 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:58:06.375977 17389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0: Generated new TSK 0
I20260807 02:58:06.377385 17389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:58:07.173240 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:45715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:40765
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:58:07 dist-test-slave-3h5h krb5kdc[16700](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786071487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 02:58:07.675868 17435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:58:07.676348 17435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:58:07.676649 17435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:58:07.691788 17435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:58:07.692056 17435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:58:07.692363 17435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:58:07.710816 17435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.121.84:45715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:40765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:58:07.712396 17435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:58:07.714143 17435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:58:07.732606 17446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:58:07.732853 17449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:58:07.735400 17435 server_base.cc:1034] running on GCE node
W20260807 02:58:07.737108 17447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:58:07.737919 17435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:58:07.739217 17435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:58:07.740484 17435 hybrid_clock.cc:648] HybridClock initialized: now 1786071487740367 us; error 87 us; skew 500 ppm
I20260807 02:58:07.740988 17435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:58:07 dist-test-slave-3h5h krb5kdc[16700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:58:07.745070 17435 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:58:07.747031 17435 webserver.cc:460] Webserver started at http://127.1.121.65:46555/ using document root <none> and password file <none>
I20260807 02:58:07.748044 17435 fs_manager.cc:359] Metadata directory not provided
I20260807 02:58:07.748236 17435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:58:07.748638 17435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:58:07 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:58:07.787586 17435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "396add1c26af4ff390f783b9232dd29e"
format_stamp: "Formatted at 2026-08-07 02:58:07 on dist-test-slave-3h5h"
server_key: "e782946ad2abc8fce4ffc51383630733"
server_key_iv: "93cbcf19abfcaa11d85c226fb5f991db"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:07.788900 17435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "396add1c26af4ff390f783b9232dd29e"
format_stamp: "Formatted at 2026-08-07 02:58:07 on dist-test-slave-3h5h"
server_key: "e782946ad2abc8fce4ffc51383630733"
server_key_iv: "93cbcf19abfcaa11d85c226fb5f991db"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:07.801167 17435 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.005s
I20260807 02:58:07.835088 17458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:07.837518 17435 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.001s
I20260807 02:58:07.837842 17435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "396add1c26af4ff390f783b9232dd29e"
format_stamp: "Formatted at 2026-08-07 02:58:07 on dist-test-slave-3h5h"
server_key: "e782946ad2abc8fce4ffc51383630733"
server_key_iv: "93cbcf19abfcaa11d85c226fb5f991db"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:07.838322 17435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:58:07.881155 17435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:58:07.887691 17435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:58:07.888100 17435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:58:07.889791 17435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:58:07.892330 17435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:58:07.892434 17435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:07.892570 17435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:58:07.892670 17435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:58:07 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:58:08.160866 17435 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:35191
I20260807 02:58:08.160868 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:35191 every 8 connection(s)
I20260807 02:58:08.162782 17435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:58:08.164258 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 17435
I20260807 02:58:08.164458 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:58:08.165362 1509 external_mini_cluster.cc:1428] Setting key cda8be40f881e2d6ced5ef39a9492d19
Aug 07 02:58:08 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
I20260807 02:58:08.215073 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:08.169223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:36963 (local address 127.1.121.126:40765)
0807 02:58:08.187285 (+ 18062us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:08.187292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:08.187323 (+ 31us) server_negotiation.cc:407] Connection header received
0807 02:58:08.187404 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:08.187410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:08.187535 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:08.190759 (+ 3224us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:58:08.190995 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:08.192281 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:08.195989 (+ 3708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:08.196919 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:08.197078 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:08.203623 (+ 6545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:58:08.203634 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:58:08.203643 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:58:08.203709 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:58:08.208618 (+ 4909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:08.209937 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:08.209948 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:08.209954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:08.210162 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:08.210890 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:08.210898 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:08.210902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:08.211350 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:58:08.211524 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:08.214395 (+ 2871us) server_negotiation.cc:299] Negotiation successful
0807 02:58:08.214661 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17892,"thread_start_us":18461,"threads_started":1}
I20260807 02:58:08.215075 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:08.167657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:40765 (local address 127.1.121.65:36963)
0807 02:58:08.169693 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0807 02:58:08.169758 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 02:58:08.172741 (+ 2983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:58:08.188243 (+ 15502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:58:08.188259 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:58:08.189178 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:58:08.190431 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:08.190471 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:08.192500 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:08.192510 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:08.195751 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:08.195772 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:08.197525 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:08.197537 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:08.198007 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:08.199237 (+ 1230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:58:08.199298 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:58:08.203275 (+ 3977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:58:08.208889 (+ 5614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:58:08.208901 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:58:08.208926 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:58:08.209730 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:58:08.210337 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:58:08.210346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:58:08.210352 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:58:08.210724 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:58:08.211627 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:58:08.211639 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:58:08.212181 (+ 542us) client_negotiation.cc:769] Sending connection context
0807 02:58:08.212606 (+ 425us) client_negotiation.cc:240] Negotiation successful
0807 02:58:08.213305 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1642,"thread_start_us":1607,"threads_started":1}
I20260807 02:58:08.217135 17612 heartbeater.cc:347] Connected to a master server at 127.1.121.126:40765
I20260807 02:58:08.217536 17612 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
I20260807 02:58:08.218662 17612 heartbeater.cc:511] Master 127.1.121.126:40765 requested a full tablet report, sending...
I20260807 02:58:08.221596 17325 ts_manager.cc:194] Registered new tserver with Master: 396add1c26af4ff390f783b9232dd29e (127.1.121.65:35191)
I20260807 02:58:08.224017 17325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:36963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:58:08.265830 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:45715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:40765
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:58:08 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
W20260807 02:58:08.823294 17618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:58:08.823751 17618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:58:08.824008 17618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:58:08.833180 17618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:58:08.833338 17618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:58:08.833515 17618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:58:08.844278 17618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.121.84:45715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:40765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:58:08.845798 17618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:58:08.847203 17618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:58:08.862344 17634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:58:08.862557 17637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:58:08.863224 17640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:58:08.864694 17618 server_base.cc:1034] running on GCE node
I20260807 02:58:08.865491 17618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:58:08.866696 17618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:58:08.868069 17618 hybrid_clock.cc:648] HybridClock initialized: now 1786071488867987 us; error 58 us; skew 500 ppm
I20260807 02:58:08.868526 17618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:58:08 dist-test-slave-3h5h krb5kdc[16700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:58:08.872053 17618 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:58:08.873857 17618 webserver.cc:460] Webserver started at http://127.1.121.66:37701/ using document root <none> and password file <none>
I20260807 02:58:08.874760 17618 fs_manager.cc:359] Metadata directory not provided
I20260807 02:58:08.874912 17618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:58:08.875288 17618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:58:08 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:58:08.893399 17618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "bb0722f779714a1fa842384084737e52"
format_stamp: "Formatted at 2026-08-07 02:58:08 on dist-test-slave-3h5h"
server_key: "7ec99f8379e2937c0e5a4eba4a019309"
server_key_iv: "34dd8a7a837c76741d2d2331b8ff12af"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:08.894876 17618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "bb0722f779714a1fa842384084737e52"
format_stamp: "Formatted at 2026-08-07 02:58:08 on dist-test-slave-3h5h"
server_key: "7ec99f8379e2937c0e5a4eba4a019309"
server_key_iv: "34dd8a7a837c76741d2d2331b8ff12af"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:08.902969 17618 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s
I20260807 02:58:08.923028 17649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:08.924696 17618 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260807 02:58:08.924945 17618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "bb0722f779714a1fa842384084737e52"
format_stamp: "Formatted at 2026-08-07 02:58:08 on dist-test-slave-3h5h"
server_key: "7ec99f8379e2937c0e5a4eba4a019309"
server_key_iv: "34dd8a7a837c76741d2d2331b8ff12af"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:08.925228 17618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:58:08.946698 17618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:58:08.952452 17618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:58:08.952950 17618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:58:08.954107 17618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:58:08.955725 17618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:58:08.955844 17618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:08.956022 17618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:58:08.956139 17618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:09.014611 17618 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:33961
I20260807 02:58:09.014626 17804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:33961 every 8 connection(s)
I20260807 02:58:09.016060 17618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:58:09.023687 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 17618
I20260807 02:58:09.023869 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:58:09.024396 1509 external_mini_cluster.cc:1428] Setting key 54e3b5a953c8b95624706490602bb923
Aug 07 02:58:09 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:58:09.042233 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.019262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:37861 (local address 127.1.121.126:40765)
0807 02:58:09.019913 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:09.019918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:09.020920 (+ 1002us) server_negotiation.cc:407] Connection header received
0807 02:58:09.022860 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:09.022866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:09.022974 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:09.023103 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:58:09.024751 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.025833 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.028291 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.029131 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.029270 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.034000 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:58:09.034009 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:58:09.034017 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:58:09.034075 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:58:09.038409 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:09.039498 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:09.039506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:09.039513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:09.039698 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:09.040188 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:09.040194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:09.040197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:09.040513 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:58:09.040682 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:09.041508 (+ 826us) server_negotiation.cc:299] Negotiation successful
0807 02:58:09.041761 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":415,"threads_started":1}
I20260807 02:58:09.043476 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.019848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:40765 (local address 127.1.121.66:37861)
0807 02:58:09.020732 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0807 02:58:09.020772 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 02:58:09.022575 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:58:09.023355 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:58:09.023364 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:58:09.023868 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:58:09.024574 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.024595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.026013 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.026019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.028117 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.028133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.029434 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.029441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.029781 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.030586 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:58:09.030615 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:58:09.033711 (+ 3096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:58:09.038595 (+ 4884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:58:09.038607 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:58:09.038628 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:58:09.039342 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:58:09.039834 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:58:09.039840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:58:09.039848 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:58:09.040058 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:58:09.040672 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:58:09.040681 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:58:09.041125 (+ 444us) client_negotiation.cc:769] Sending connection context
0807 02:58:09.041415 (+ 290us) client_negotiation.cc:240] Negotiation successful
0807 02:58:09.041992 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":381,"threads_started":1}
I20260807 02:58:09.045349 17805 heartbeater.cc:347] Connected to a master server at 127.1.121.126:40765
I20260807 02:58:09.045696 17805 heartbeater.cc:464] Registering TS with master...
I20260807 02:58:09.046495 17805 heartbeater.cc:511] Master 127.1.121.126:40765 requested a full tablet report, sending...
I20260807 02:58:09.048131 17325 ts_manager.cc:194] Registered new tserver with Master: bb0722f779714a1fa842384084737e52 (127.1.121.66:33961)
I20260807 02:58:09.049091 17325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:37861
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:58:09.109308 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:45715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:40765
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:58:09.235791 17612 heartbeater.cc:503] Master 127.1.121.126:40765 was elected leader, sending a full tablet report...
W20260807 02:58:09.527408 17815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:58:09.527735 17815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:58:09.527921 17815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:58:09.535712 17815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:58:09.535825 17815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:58:09.535991 17815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:58:09.546631 17815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.121.84:45715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:40765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:58:09.547936 17815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:58:09.549374 17815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:58:09.563649 17822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:58:09.563599 17825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:58:09.563575 17823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:58:09.564484 17815 server_base.cc:1034] running on GCE node
I20260807 02:58:09.565119 17815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:58:09.566110 17815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:58:09.567329 17815 hybrid_clock.cc:648] HybridClock initialized: now 1786071489567289 us; error 41 us; skew 500 ppm
I20260807 02:58:09.567631 17815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:58:09 dist-test-slave-3h5h krb5kdc[16700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:58:09.571391 17815 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:58:09.573222 17815 webserver.cc:460] Webserver started at http://127.1.121.67:35595/ using document root <none> and password file <none>
I20260807 02:58:09.573931 17815 fs_manager.cc:359] Metadata directory not provided
I20260807 02:58:09.574036 17815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:58:09.574358 17815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:58:09 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:58:09.593905 17815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "59ba32a628ed4736811be4047c7149bc"
format_stamp: "Formatted at 2026-08-07 02:58:09 on dist-test-slave-3h5h"
server_key: "4301bbbf52a49784be29ec04e297bc0f"
server_key_iv: "05f4a9e19fb5efc62b4e78594cc4f43e"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:09.595017 17815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "59ba32a628ed4736811be4047c7149bc"
format_stamp: "Formatted at 2026-08-07 02:58:09 on dist-test-slave-3h5h"
server_key: "4301bbbf52a49784be29ec04e297bc0f"
server_key_iv: "05f4a9e19fb5efc62b4e78594cc4f43e"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:09.602778 17815 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s
I20260807 02:58:09.621729 17832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:09.623528 17815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:58:09.623853 17815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "59ba32a628ed4736811be4047c7149bc"
format_stamp: "Formatted at 2026-08-07 02:58:09 on dist-test-slave-3h5h"
server_key: "4301bbbf52a49784be29ec04e297bc0f"
server_key_iv: "05f4a9e19fb5efc62b4e78594cc4f43e"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:09.624339 17815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:58:09.663756 17815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:58:09.668711 17815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:58:09.669267 17815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:58:09.670495 17815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:58:09.672600 17815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:58:09.672739 17815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:09.672961 17815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:58:09.673058 17815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:09.720942 17815 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:42771
I20260807 02:58:09.721079 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:42771 every 8 connection(s)
I20260807 02:58:09.722361 17815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:58:09.723552 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 17815
I20260807 02:58:09.723789 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:58:09.724426 1509 external_mini_cluster.cc:1428] Setting key 692b9195788ebdae9403c62ec8bd9625
Aug 07 02:58:09 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071489, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:58:09.746833 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.725772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:57227 (local address 127.1.121.126:40765)
0807 02:58:09.726637 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:09.726643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:09.727753 (+ 1110us) server_negotiation.cc:407] Connection header received
0807 02:58:09.729945 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:09.729949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:09.730013 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:09.730138 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:58:09.732396 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.733229 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.735211 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.735718 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.735818 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.740125 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:58:09.740130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:58:09.740133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:58:09.740194 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:58:09.743462 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:09.744263 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:09.744268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:09.744270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:09.744405 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:09.744827 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:09.744901 (+ 74us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:09.744907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:09.745353 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:58:09.745486 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:09.746232 (+ 746us) server_negotiation.cc:299] Negotiation successful
0807 02:58:09.746537 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":429,"threads_started":1}
I20260807 02:58:09.747795 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.726653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:40765 (local address 127.1.121.67:57227)
0807 02:58:09.727583 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0807 02:58:09.727638 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 02:58:09.729702 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:58:09.730529 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:58:09.730539 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:58:09.731151 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:58:09.732204 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.732228 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.733420 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.733425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.735062 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.735078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.736073 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.736082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.736373 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.737208 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:58:09.737233 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:58:09.739898 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:58:09.743610 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:58:09.743617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:58:09.743632 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:58:09.744124 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:58:09.744520 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:58:09.744524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:58:09.744526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:58:09.744726 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:58:09.745488 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:58:09.745494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:58:09.745831 (+ 337us) client_negotiation.cc:769] Sending connection context
0807 02:58:09.746119 (+ 288us) client_negotiation.cc:240] Negotiation successful
0807 02:58:09.746594 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":413,"threads_started":1}
I20260807 02:58:09.749632 17991 heartbeater.cc:347] Connected to a master server at 127.1.121.126:40765
I20260807 02:58:09.749962 17991 heartbeater.cc:464] Registering TS with master...
I20260807 02:58:09.750811 17991 heartbeater.cc:511] Master 127.1.121.126:40765 requested a full tablet report, sending...
I20260807 02:58:09.752787 17325 ts_manager.cc:194] Registered new tserver with Master: 59ba32a628ed4736811be4047c7149bc (127.1.121.67:42771)
I20260807 02:58:09.753957 17325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:57227
I20260807 02:58:09.754552 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:58:09.774165 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.759974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.1.121.126:40765)
0807 02:58:09.760200 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:09.760207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:09.760736 (+ 529us) server_negotiation.cc:407] Connection header received
0807 02:58:09.761046 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:09.761079 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:09.761161 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:09.761250 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:58:09.762767 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.763539 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.764892 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.765453 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.765556 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.767960 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:58:09.767965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:58:09.767968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:58:09.767998 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:58:09.771049 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:09.771740 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:09.771744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:09.771746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:09.771888 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:09.772492 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:09.772497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:09.772499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:09.772798 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:58:09.773127 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:09.773526 (+ 399us) server_negotiation.cc:299] Negotiation successful
0807 02:58:09.773768 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 02:58:09.778497 17325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:58:09.786712 17325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:58:09.813844 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.801015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.1.121.66:33961)
0807 02:58:09.804271 (+ 3256us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:09.804281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:09.804306 (+ 25us) server_negotiation.cc:407] Connection header received
0807 02:58:09.804411 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:09.804422 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:09.804709 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:09.805946 (+ 1237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:58:09.806385 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.807715 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.810986 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.812561 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.812738 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.812986 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:09.813281 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 02:58:09.813459 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3099,"thread_start_us":3262,"threads_started":1}
I20260807 02:58:09.813900 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.800510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:33961 (local address 127.0.0.1:42914)
0807 02:58:09.801689 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0807 02:58:09.801739 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 02:58:09.802004 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:58:09.805043 (+ 3039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:58:09.805053 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:58:09.805091 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:58:09.806061 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.806074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.807845 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.807858 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.810772 (+ 2914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.810793 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.812802 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.812809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.813036 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.813092 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 02:58:09.813237 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 02:58:09.813426 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"spinlock_wait_cycles":4992,"thread_start_us":717,"threads_started":1}
I20260807 02:58:09.815840 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.800778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.1.121.67:42771)
0807 02:58:09.803049 (+ 2271us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:09.803061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:09.804247 (+ 1186us) server_negotiation.cc:407] Connection header received
0807 02:58:09.804551 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:09.804557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:09.804734 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:09.804938 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:58:09.808001 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.809637 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.812276 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.813837 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.814008 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.814247 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:09.814902 (+ 655us) server_negotiation.cc:299] Negotiation successful
0807 02:58:09.815263 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2061,"thread_start_us":2256,"threads_started":1}
I20260807 02:58:09.816128 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.802247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:35191 (local address 127.0.0.1:52090)
0807 02:58:09.805025 (+ 2778us) negotiation.cc:107] Waiting for socket to connect
0807 02:58:09.805039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:58:09.805192 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:58:09.805734 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:58:09.805739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:58:09.805764 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:58:09.808243 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.808255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.809895 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.809900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.812369 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.812384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.815070 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.815077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.815258 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.815289 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 02:58:09.815385 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 02:58:09.815679 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2645,"mutex_wait_us":28,"thread_start_us":1854,"threads_started":1}
I20260807 02:58:09.816671 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.800541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:42771 (local address 127.0.0.1:34354)
0807 02:58:09.804129 (+ 3588us) negotiation.cc:107] Waiting for socket to connect
0807 02:58:09.804148 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:58:09.804363 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:58:09.805331 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:58:09.805336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:58:09.805381 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:58:09.807708 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.807725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.809801 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.809808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.812101 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:09.812114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.814453 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.814460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:09.814674 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.814707 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 02:58:09.816204 (+ 1497us) client_negotiation.cc:240] Negotiation successful
0807 02:58:09.816355 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3464,"thread_start_us":4210,"threads_started":1}
I20260807 02:58:09.816989 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:09.802827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.1.121.65:35191)
0807 02:58:09.803841 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:09.803853 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:09.805144 (+ 1291us) server_negotiation.cc:407] Connection header received
0807 02:58:09.805370 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:09.805375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:09.805532 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:09.805725 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:58:09.808454 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.809706 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.812539 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:09.814463 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:09.814633 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:09.814870 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:09.816422 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0807 02:58:09.816610 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":610,"threads_started":1}
I20260807 02:58:09.818112 17719 tablet_service.cc:1467] Processing CreateTablet for tablet 0940a09fb8dc422ea2df7809ac0a98af (DEFAULT_TABLE table=test-table [id=5bb3cdc0e4504793868a9c6e3a89e29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:58:09.819792 17719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0940a09fb8dc422ea2df7809ac0a98af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:58:09.820801 17525 tablet_service.cc:1467] Processing CreateTablet for tablet 0940a09fb8dc422ea2df7809ac0a98af (DEFAULT_TABLE table=test-table [id=5bb3cdc0e4504793868a9c6e3a89e29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:58:09.822630 17525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0940a09fb8dc422ea2df7809ac0a98af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:58:09.822275 17902 tablet_service.cc:1467] Processing CreateTablet for tablet 0940a09fb8dc422ea2df7809ac0a98af (DEFAULT_TABLE table=test-table [id=5bb3cdc0e4504793868a9c6e3a89e29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:58:09.823489 17902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0940a09fb8dc422ea2df7809ac0a98af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:58:09.836505 18023 tablet_bootstrap.cc:492] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: Bootstrap starting.
I20260807 02:58:09.840241 18023 tablet_bootstrap.cc:654] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: Neither blocks nor log segments found. Creating new log.
I20260807 02:58:09.841297 18024 tablet_bootstrap.cc:492] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: Bootstrap starting.
I20260807 02:58:09.842572 18023 log.cc:824] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: Log is configured to *not* fsync() on all Append() calls
I20260807 02:58:09.845969 18025 tablet_bootstrap.cc:492] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: Bootstrap starting.
I20260807 02:58:09.846585 18024 tablet_bootstrap.cc:654] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: Neither blocks nor log segments found. Creating new log.
I20260807 02:58:09.848450 18024 log.cc:824] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: Log is configured to *not* fsync() on all Append() calls
I20260807 02:58:09.850294 18025 tablet_bootstrap.cc:654] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: Neither blocks nor log segments found. Creating new log.
I20260807 02:58:09.851567 18023 tablet_bootstrap.cc:492] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: No bootstrap required, opened a new log
I20260807 02:58:09.851915 18023 ts_tablet_manager.cc:1397] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260807 02:58:09.852223 18025 log.cc:824] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:58:09.852420 18024 tablet_bootstrap.cc:492] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: No bootstrap required, opened a new log
I20260807 02:58:09.852797 18024 ts_tablet_manager.cc:1397] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260807 02:58:09.857558 18025 tablet_bootstrap.cc:492] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: No bootstrap required, opened a new log
I20260807 02:58:09.857867 18025 ts_tablet_manager.cc:1397] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260807 02:58:09.861436 18023 raft_consensus.cc:348] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } }
I20260807 02:58:09.861838 18023 raft_consensus.cc:374] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:58:09.861965 18023 raft_consensus.cc:729] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ba32a628ed4736811be4047c7149bc, State: Initialized, Role: FOLLOWER
I20260807 02:58:09.862630 18023 consensus_queue.cc:260] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } }
I20260807 02:58:09.863914 17991 heartbeater.cc:503] Master 127.1.121.126:40765 was elected leader, sending a full tablet report...
I20260807 02:58:09.864614 18023 ts_tablet_manager.cc:1428] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260807 02:58:09.866701 18024 raft_consensus.cc:348] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } }
I20260807 02:58:09.867242 18024 raft_consensus.cc:374] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:58:09.867369 18024 raft_consensus.cc:729] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0722f779714a1fa842384084737e52, State: Initialized, Role: FOLLOWER
I20260807 02:58:09.868172 18024 consensus_queue.cc:260] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } }
I20260807 02:58:09.869887 17805 heartbeater.cc:503] Master 127.1.121.126:40765 was elected leader, sending a full tablet report...
I20260807 02:58:09.871109 18024 ts_tablet_manager.cc:1428] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260807 02:58:09.872252 18025 raft_consensus.cc:348] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } }
I20260807 02:58:09.872674 18025 raft_consensus.cc:374] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:58:09.872798 18025 raft_consensus.cc:729] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396add1c26af4ff390f783b9232dd29e, State: Initialized, Role: FOLLOWER
I20260807 02:58:09.873636 18025 consensus_queue.cc:260] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } }
I20260807 02:58:09.875844 18025 ts_tablet_manager.cc:1428] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
W20260807 02:58:09.926333 17613 tablet.cc:2369] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:58:09.975139 17992 tablet.cc:2369] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:58:10.018864 17806 tablet.cc:2369] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:58:10.056874 18034 raft_consensus.cc:482] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:58:10.057263 18034 raft_consensus.cc:504] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } }
I20260807 02:58:10.059129 18034 leader_election.cc:288] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59ba32a628ed4736811be4047c7149bc (127.1.121.67:42771), bb0722f779714a1fa842384084737e52 (127.1.121.66:33961)
I20260807 02:58:10.067999 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:10.059663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:42771 (local address 127.1.121.65:50387)
0807 02:58:10.060675 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0807 02:58:10.060694 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:58:10.060911 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:58:10.061266 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:58:10.061270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:58:10.061301 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:58:10.061829 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:10.061843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.063491 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.063496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:10.065358 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:10.065371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.067093 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.067101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:10.067293 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:10.067328 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 02:58:10.067434 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 02:58:10.067544 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"thread_start_us":580,"threads_started":1}
I20260807 02:58:10.068306 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:10.060250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:50387 (local address 127.1.121.67:42771)
0807 02:58:10.060490 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:10.060497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:10.060790 (+ 293us) server_negotiation.cc:407] Connection header received
0807 02:58:10.061062 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:10.061072 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:10.061134 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:10.061242 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:58:10.062034 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.063349 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.065576 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.066827 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.066996 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:10.067060 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:10.067502 (+ 442us) server_negotiation.cc:299] Negotiation successful
0807 02:58:10.067650 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 02:58:10.068439 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:10.059843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:54695 (local address 127.1.121.66:33961)
0807 02:58:10.060084 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:10.060091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:10.061711 (+ 1620us) server_negotiation.cc:407] Connection header received
0807 02:58:10.061844 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:10.061850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:10.061960 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:10.062073 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:58:10.062930 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.063818 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.066178 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.067170 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.067318 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:10.067387 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:10.068039 (+ 652us) server_negotiation.cc:299] Negotiation successful
0807 02:58:10.068189 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:58:10.068686 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:10.060522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:33961 (local address 127.1.121.65:54695)
0807 02:58:10.061592 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0807 02:58:10.061609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:58:10.061727 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:58:10.062146 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:58:10.062153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:58:10.062186 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:58:10.062742 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:10.062758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.063977 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.063981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:10.065993 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:58:10.066013 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.067479 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.067536 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:58:10.067830 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:10.067870 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 02:58:10.067985 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 02:58:10.068143 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":66,"thread_start_us":613,"threads_started":1}
I20260807 02:58:10.069571 17929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940a09fb8dc422ea2df7809ac0a98af" candidate_uuid: "396add1c26af4ff390f783b9232dd29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ba32a628ed4736811be4047c7149bc" is_pre_election: true
I20260807 02:58:10.070065 17748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940a09fb8dc422ea2df7809ac0a98af" candidate_uuid: "396add1c26af4ff390f783b9232dd29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0722f779714a1fa842384084737e52" is_pre_election: true
I20260807 02:58:10.070338 17929 raft_consensus.cc:2454] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 396add1c26af4ff390f783b9232dd29e in term 0.
I20260807 02:58:10.070761 17748 raft_consensus.cc:2454] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 396add1c26af4ff390f783b9232dd29e in term 0.
I20260807 02:58:10.071244 17462 leader_election.cc:302] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 396add1c26af4ff390f783b9232dd29e, 59ba32a628ed4736811be4047c7149bc; no voters:
I20260807 02:58:10.071798 18034 raft_consensus.cc:2781] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:58:10.071979 18034 raft_consensus.cc:482] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:58:10.072099 18034 raft_consensus.cc:3037] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:58:10.074621 18034 raft_consensus.cc:504] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } }
I20260807 02:58:10.075372 18034 leader_election.cc:288] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [CANDIDATE]: Term 1 election: Requested vote from peers 59ba32a628ed4736811be4047c7149bc (127.1.121.67:42771), bb0722f779714a1fa842384084737e52 (127.1.121.66:33961)
I20260807 02:58:10.076074 17748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940a09fb8dc422ea2df7809ac0a98af" candidate_uuid: "396add1c26af4ff390f783b9232dd29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0722f779714a1fa842384084737e52"
I20260807 02:58:10.076092 17929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0940a09fb8dc422ea2df7809ac0a98af" candidate_uuid: "396add1c26af4ff390f783b9232dd29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ba32a628ed4736811be4047c7149bc"
I20260807 02:58:10.076368 17929 raft_consensus.cc:3037] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:58:10.076349 17748 raft_consensus.cc:3037] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:58:10.078801 17929 raft_consensus.cc:2454] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 396add1c26af4ff390f783b9232dd29e in term 1.
I20260807 02:58:10.079412 17462 leader_election.cc:302] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 396add1c26af4ff390f783b9232dd29e, 59ba32a628ed4736811be4047c7149bc; no voters:
I20260807 02:58:10.079433 17748 raft_consensus.cc:2454] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 396add1c26af4ff390f783b9232dd29e in term 1.
I20260807 02:58:10.079877 18034 raft_consensus.cc:2781] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:58:10.080690 18034 raft_consensus.cc:686] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 1 LEADER]: Becoming Leader. State: Replica: 396add1c26af4ff390f783b9232dd29e, State: Running, Role: LEADER
I20260807 02:58:10.081251 18034 consensus_queue.cc:237] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } }
I20260807 02:58:10.087092 17325 catalog_manager.cc:5495] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e reported cstate change: term changed from 0 to 1, leader changed from <none> to 396add1c26af4ff390f783b9232dd29e (127.1.121.65). New cstate: current_term: 1 leader_uuid: "396add1c26af4ff390f783b9232dd29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "396add1c26af4ff390f783b9232dd29e" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 35191 } health_report { overall_health: HEALTHY } } }
I20260807 02:58:10.118657 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:10.111400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.1.121.65:35191)
0807 02:58:10.111693 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:10.111703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:10.111757 (+ 54us) server_negotiation.cc:407] Connection header received
0807 02:58:10.112059 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:10.112067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:10.112186 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:10.112374 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:58:10.112819 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.113955 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.115749 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.116250 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.116413 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:10.117237 (+ 824us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:58:10.117920 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:58:10.118122 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:10.118187 (+ 65us) server_negotiation.cc:299] Negotiation successful
0807 02:58:10.118277 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260807 02:58:10.129674 17929 raft_consensus.cc:1261] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [term 1 FOLLOWER]: Refusing update from remote peer 396add1c26af4ff390f783b9232dd29e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:58:10.129673 17748 raft_consensus.cc:1261] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [term 1 FOLLOWER]: Refusing update from remote peer 396add1c26af4ff390f783b9232dd29e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:58:10.130836 18034 consensus_queue.cc:1035] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0722f779714a1fa842384084737e52" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:58:10.131171 18038 consensus_queue.cc:1035] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ba32a628ed4736811be4047c7149bc" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:58:10.164083 17324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:58:10.164616 17324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:58:10.169597 17324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c52f59be554643da84798d2ee65d1fb0: Sending DeleteTablet for 3 replicas of tablet 0940a09fb8dc422ea2df7809ac0a98af
I20260807 02:58:10.171528 17525 tablet_service.cc:1514] Processing DeleteTablet for tablet 0940a09fb8dc422ea2df7809ac0a98af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:58:10 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:52090
I20260807 02:58:10.171614 17902 tablet_service.cc:1514] Processing DeleteTablet for tablet 0940a09fb8dc422ea2df7809ac0a98af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:58:10 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:34354
I20260807 02:58:10.172956 17719 tablet_service.cc:1514] Processing DeleteTablet for tablet 0940a09fb8dc422ea2df7809ac0a98af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:58:10 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:42914
I20260807 02:58:10.173420 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 17435
I20260807 02:58:10.178651 18056 tablet_replica.cc:330] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: stopping tablet replica
I20260807 02:58:10.180585 18056 raft_consensus.cc:2229] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:58:10.181895 18055 tablet_replica.cc:330] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: stopping tablet replica
I20260807 02:58:10.182339 18055 raft_consensus.cc:2229] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:58:10.183171 18055 raft_consensus.cc:2258] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:58:10.186205 18055 ts_tablet_manager.cc:1905] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:58:10.187731 18058 tablet_replica.cc:330] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: stopping tablet replica
I20260807 02:58:10.188182 18058 raft_consensus.cc:2229] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:58:10.188730 18058 raft_consensus.cc:2258] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:58:10.190445 18056 raft_consensus.cc:2258] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:58:10.191048 18058 ts_tablet_manager.cc:1905] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:58:10.193150 18056 ts_tablet_manager.cc:1905] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:58:10.195667 18055 ts_tablet_manager.cc:1918] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:58:10.195940 18055 log.cc:1196] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/0940a09fb8dc422ea2df7809ac0a98af
Aug 07 02:58:10 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:58:10.196720 18055 ts_tablet_manager.cc:1939] T 0940a09fb8dc422ea2df7809ac0a98af P 396add1c26af4ff390f783b9232dd29e: Deleting consensus metadata
I20260807 02:58:10.199107 17308 catalog_manager.cc:4841] TS 396add1c26af4ff390f783b9232dd29e (127.1.121.65:35191): tablet 0940a09fb8dc422ea2df7809ac0a98af (table test-table [id=5bb3cdc0e4504793868a9c6e3a89e29a]) successfully deleted
I20260807 02:58:10.200150 18058 ts_tablet_manager.cc:1918] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:58:10.200356 18058 log.cc:1196] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/0940a09fb8dc422ea2df7809ac0a98af
I20260807 02:58:10.201165 18058 ts_tablet_manager.cc:1939] T 0940a09fb8dc422ea2df7809ac0a98af P bb0722f779714a1fa842384084737e52: Deleting consensus metadata
I20260807 02:58:10.201465 18056 ts_tablet_manager.cc:1918] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:58:10.201673 18056 log.cc:1196] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/0940a09fb8dc422ea2df7809ac0a98af
I20260807 02:58:10.202205 18056 ts_tablet_manager.cc:1939] T 0940a09fb8dc422ea2df7809ac0a98af P 59ba32a628ed4736811be4047c7149bc: Deleting consensus metadata
I20260807 02:58:10.203315 17308 catalog_manager.cc:4841] TS bb0722f779714a1fa842384084737e52 (127.1.121.66:33961): tablet 0940a09fb8dc422ea2df7809ac0a98af (table test-table [id=5bb3cdc0e4504793868a9c6e3a89e29a]) successfully deleted
I20260807 02:58:10.204144 17304 catalog_manager.cc:4841] TS 59ba32a628ed4736811be4047c7149bc (127.1.121.67:42771): tablet 0940a09fb8dc422ea2df7809ac0a98af (table test-table [id=5bb3cdc0e4504793868a9c6e3a89e29a]) successfully deleted
I20260807 02:58:10.206938 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:10.174799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.1.121.65:35191)
0807 02:58:10.175371 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:10.175381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:10.181010 (+ 5629us) server_negotiation.cc:407] Connection header received
0807 02:58:10.183580 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:10.183587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:10.183681 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:10.183771 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:58:10.185979 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.187459 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.190614 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.191397 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.191535 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:10.198506 (+ 6971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:58:10.198526 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:58:10.198549 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:58:10.198610 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:58:10.203197 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:10.204258 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:10.204271 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:10.204275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:10.204462 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:10.204999 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:10.205004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:10.205006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:10.205407 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:58:10.205753 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:10.206170 (+ 417us) server_negotiation.cc:299] Negotiation successful
0807 02:58:10.206454 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409}
I20260807 02:58:10.207710 17603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:58:10.311020 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 17435
I20260807 02:58:10.337206 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 17618
Aug 07 02:58:10 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:58:10.353056 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:10.337973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.1.121.66:33961)
0807 02:58:10.338271 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:10.338276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:10.338297 (+ 21us) server_negotiation.cc:407] Connection header received
0807 02:58:10.338473 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:10.338480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:10.338556 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:10.338675 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:58:10.340114 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.340898 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.342218 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.342693 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.342837 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:10.347123 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:58:10.347138 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:58:10.347154 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:58:10.347188 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:58:10.350105 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:10.350827 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:10.350836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:10.350839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:10.350969 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:10.351357 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:10.351362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:10.351366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:10.351749 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:58:10.352087 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:10.352376 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 02:58:10.352589 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260807 02:58:10.353732 17799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:58:10.446036 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 17618
I20260807 02:58:10.472878 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 17815
Aug 07 02:58:10 dist-test-slave-3h5h krb5kdc[16700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:58:10.490450 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:10.473860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.1.121.67:42771)
0807 02:58:10.474067 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:10.474073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:10.474097 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:58:10.474290 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:10.474295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:10.474379 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:10.474502 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:58:10.475926 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.476897 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.478200 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:10.478698 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:10.478807 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:10.483466 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:58:10.483487 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:58:10.483507 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:58:10.483557 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:58:10.487457 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:10.488230 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:10.488238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:10.488241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:10.488398 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:10.488768 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:10.488772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:10.488774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:10.489349 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:58:10.489608 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:10.489861 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 02:58:10.490124 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 02:58:10.491240 17986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:58:10.582391 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 17815
I20260807 02:58:10.607744 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 17282
I20260807 02:58:10.609112 17365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:58:10.723198 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 17282
I20260807 02:58:10.755944 1509 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 02:58:11.102080 1509 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 02:58:11.102411 1509 mini_ranger.cc:66] Stopping Ranger...
I20260807 02:58:11.458493 1509 mini_ranger.cc:68] Stopped Ranger
2026-08-07 02:58:11.458 UTC [16739] LOG: received smart shutdown request
2026-08-07 02:58:11.463 UTC [16739] LOG: background worker "logical replication launcher" (PID 16748) exited with exit code 1
2026-08-07 02:58:11.463 UTC [16743] LOG: shutting down
2026-08-07 02:58:11.674 UTC [16739] LOG: database system is shut down
2026-08-07T02:58:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55374 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:58:11 dist-test-slave-3h5h krb5kdc[18080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:58:11 dist-test-slave-3h5h krb5kdc[18080](info): set up 2 sockets
Aug 07 02:58:11 dist-test-slave-3h5h krb5kdc[18080](info): commencing operation
krb5kdc: starting...
W20260807 02:58:13.161705 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.009s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:58:13 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:58:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.121.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.121.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/rangeradmin_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/rangeradmin_127.1.121.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.121.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.121.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/rangerlookup_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/rangerlookup_127.1.121.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.121.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.121.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/HTTP_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/HTTP_127.1.121.84@KRBTEST.COM.keytab.
I20260807 02:58:13.612183 1509 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taska5bS7K/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/postgres -l logfile start
2026-08-07 02:58:16.385 UTC [18119] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 02:58:16.385 UTC [18119] LOG: listening on IPv4 address "127.1.121.84", port 54445
2026-08-07 02:58:16.389 UTC [18119] LOG: listening on Unix socket "/tmp/.s.PGSQL.54445"
2026-08-07 02:58:16.414 UTC [18121] LOG: database system was shut down at 2026-08-07 02:58:14 UTC
2026-08-07 02:58:16.418 UTC [18119] LOG: database system is ready to accept connections
I20260807 02:58:17.577822 1509 mini_postgres.cc:96] Postgres bound to 54445
2026-08-07 02:58:17.602 UTC [18134] FATAL: database "slave" does not exist
127.1.121.84:54445 - accepting connections
I20260807 02:58:17.603168 1509 mini_ranger.cc:161] Starting Ranger...
I20260807 02:58:17.636029 1509 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 02:58:18.193742 1509 mini_ranger.cc:87] Created ranger Postgres database
I20260807 02:58:18.193883 1509 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-admin
2026-08-07 02:58:18,507 [I] DB FLAVOR :POSTGRES
2026-08-07 02:58:18,508 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 02:58:18,508 [I] Checking connection..
2026-08-07 02:58:18,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 02:58:18,775 [I] Checking connection passed.
2026-08-07 02:58:18,776 [I] --------- Verifying version history table ---------
2026-08-07 02:58:18,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:58:19,093 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 02:58:19,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:58:19,355 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 02:58:19,355 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 02:58:19,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 02:58:19.612 UTC [18224] WARNING: there is no transaction in progress
2026-08-07 02:58:19,620 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 02:58:19,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:58:19,919 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 02:58:19,919 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 02:58:19,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 02:58:20,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 02:58:20,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3h5h.c.gcp-upstream.internal','N') ;"
2026-08-07 02:58:20,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 02:58:21,023 [I] Table x_portal_user does not exist in database ranger
2026-08-07 02:58:21,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 02:58:21,281 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 02:58:21,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 02:58:21,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 02:58:21,821 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 02:58:21,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 02:58:22.813 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:22.829 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:22.846 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:22.862 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:22.881 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:22.902 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:22.971 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:22.978 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:22.994 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:23.005 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:23.016 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:23.031 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:23.042 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:23.050 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:23.058 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:23.065 UTC [18371] WARNING: there is no transaction in progress
2026-08-07 02:58:23,868 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 02:58:23,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 02:58:24,265 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 02:58:24,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 02:58:24,536 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 02:58:24,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 02:58:24,815 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 02:58:24,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 02:58:25,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 02:58:25,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3h5h.c.gcp-upstream.internal';"
2026-08-07 02:58:25,596 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 02:58:25,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 02:58:25,866 [I] DB_PATCHES have already been applied
I20260807 02:58:25.872884 1509 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-admin:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taska5bS7K/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 02:58:25.873008 1509 mini_ranger.cc:193] Using host: 127.1.121.84
I20260807 02:58:25.896664 1509 mini_ranger.cc:227] Ranger admin URL: http://127.1.121.84:45377
Aug 07, 2026 2:58:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 2:58:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 2:58:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 2:58:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 2:58:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 2:58:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 2:58:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 02:58:27.302891 1509 mini_ranger.cc:160] Time spent starting Ranger: real 9.700s user 0.008s sys 0.061s
Aug 07, 2026 2:58:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 2:58:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 2:58:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 2:58:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 2:58:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 02:58:49.702932 1509 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.121.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.121.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/rangerkms_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.121.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/rangerkms_127.1.121.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.121.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.121.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.121.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/HTTP_127.1.121.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.121.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/HTTP_127.1.121.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 02:58:49 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071529, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 02:58:49.951644 1509 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 02:58:49.981323 1509 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 02:58:50.935660 1509 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 02:58:50.935810 1509 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-kms
I20260807 02:58:50.935863 1509 mini_ranger_kms.cc:227] Using postgres at 127.1.121.84:54445
2026-08-07 02:58:51,333 [I] DB FLAVOR :POSTGRES
2026-08-07 02:58:51,334 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 02:58:51,334 [I] Checking connection
2026-08-07 02:58:51,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 02:58:51,589 [I] connection success
2026-08-07 02:58:51,590 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 02:58:51,590 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 02:58:51,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 02:58:51,825 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 02:58:51,825 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 02:58:51,826 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 02:58:51,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.121.84:54445/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 02:58:52,098 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 02:58:52.657346 1509 mini_ranger_kms.cc:310] Created kms service
I20260807 02:58:52.874155 1509 mini_ranger_kms.cc:326] Created kudu user
I20260807 02:58:52.923388 1509 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 02:58:52 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071529, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:58:53.375543 1509 mini_ranger_kms.cc:384] Added ranger policy
I20260807 02:58:53.375916 1509 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-kms:/tmp/dist-test-taska5bS7K/build/asan/bin/postgresql.jar:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taska5bS7K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 02:58:53.376014 1509 mini_ranger_kms.cc:240] Using host: 127.1.121.84
I20260807 02:58:53.403786 1509 mini_ranger_kms.cc:276] Ranger KMS PID: 18622
I20260807 02:58:53.403945 1509 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.121.84:42229
Aug 07, 2026 2:58:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 2:58:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 2:58:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 2:58:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 2:58:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 2:58:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 02:58:55 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786071535, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.121.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 02:58:56 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071535, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.121.84@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:58:56.267287 1509 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.316s user 0.094s sys 0.340s
I20260807 02:58:56.267565 1509 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 02:58:56.267812 1509 mini_ranger_kms.cc:401] 127.1.121.84:42229/kms/v1/keys
Aug 07 02:58:56 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071535, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.121.84@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
Aug 07 02:58:58 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:58:59.012705 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41153
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:41153
--ranger_config_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:42229/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:58:59.379209 18690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:58:59.379515 18690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:58:59.379676 18690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:58:59.387311 18690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:58:59.387449 18690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:58:59.387524 18690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:58:59.387625 18690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:58:59.387709 18690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:58:59.398511 18690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.121.84:42229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:41153
--ranger_config_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:58:59.399947 18690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:58:59.401448 18690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:58:59.415366 18704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:58:59.415824 18698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:58:59.415395 18699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:58:59.415869 18690 server_base.cc:1034] running on GCE node
I20260807 02:58:59.417150 18690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:58:59.418633 18690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:58:59.419883 18690 hybrid_clock.cc:648] HybridClock initialized: now 1786071539419833 us; error 39 us; skew 500 ppm
I20260807 02:58:59.420266 18690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:58:59 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071539, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:58:59.423741 18690 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:58:59.425424 18690 webserver.cc:460] Webserver started at http://127.1.121.126:41657/ using document root <none> and password file <none>
I20260807 02:58:59.426112 18690 fs_manager.cc:359] Metadata directory not provided
I20260807 02:58:59.426247 18690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:58:59.426590 18690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:58:59 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071539, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:58:59.477793 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "ba162cb2f3b24ee79b23eefe6d90b62c"
format_stamp: "Formatted at 2026-08-07 02:58:59 on dist-test-slave-3h5h"
server_key: "a3b27b1a226e07f5acecc6d9bd698008"
server_key_iv: "ae4af33fadf0151c195ae2a7a4aee4cc"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:59.478693 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "ba162cb2f3b24ee79b23eefe6d90b62c"
format_stamp: "Formatted at 2026-08-07 02:58:59 on dist-test-slave-3h5h"
server_key: "a3b27b1a226e07f5acecc6d9bd698008"
server_key_iv: "ae4af33fadf0151c195ae2a7a4aee4cc"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:59.484727 18690 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 02:58:59.502403 18716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:58:59.504093 18690 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260807 02:58:59.504339 18690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "ba162cb2f3b24ee79b23eefe6d90b62c"
format_stamp: "Formatted at 2026-08-07 02:58:59 on dist-test-slave-3h5h"
server_key: "a3b27b1a226e07f5acecc6d9bd698008"
server_key_iv: "ae4af33fadf0151c195ae2a7a4aee4cc"
server_key_version: "kuduclusterkey@0"
I20260807 02:58:59.504595 18690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:58:59.535264 18690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:58:59.539919 18690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:58:59.540323 18690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:58:59.561589 18690 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:41153
I20260807 02:58:59.561610 18788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:41153 every 8 connection(s)
I20260807 02:58:59.563002 18690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:58:59.565842 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 18690
I20260807 02:58:59.566078 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:58:59.566785 1509 external_mini_cluster.cc:1428] Setting key 8998513008442ddf86c6ecf39743aa22
I20260807 02:58:59.567574 18789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 02:58:59 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:58:59.583782 18789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Bootstrap starting.
I20260807 02:58:59.588729 18789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Neither blocks nor log segments found. Creating new log.
I20260807 02:58:59.590271 18789 log.cc:824] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:58:59.593161 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:58:59.568924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.1.121.126:41153)
0807 02:58:59.569922 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0807 02:58:59.569951 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0807 02:58:59.570010 (+ 59us) server_negotiation.cc:407] Connection header received
0807 02:58:59.571931 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:58:59.571976 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:58:59.572572 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:58:59.573085 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:58:59.574332 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:59.576531 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:59.577972 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:58:59.579075 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:58:59.579284 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:58:59.583659 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:58:59.583671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:58:59.583692 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:58:59.583742 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:58:59.588000 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:59.588741 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:59.588753 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:59.588765 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:59.589038 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:58:59.589459 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:58:59.589465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:58:59.589469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:58:59.590251 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:58:59.590571 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 02:58:59.591130 (+ 559us) server_negotiation.cc:299] Negotiation successful
0807 02:58:59.591597 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":447,"threads_started":1}
I20260807 02:58:59.594244 18789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: No bootstrap required, opened a new log
I20260807 02:58:59.602731 18789 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } }
I20260807 02:58:59.603149 18789 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:58:59.603237 18789 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba162cb2f3b24ee79b23eefe6d90b62c, State: Initialized, Role: FOLLOWER
I20260807 02:58:59.603875 18789 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } }
I20260807 02:58:59.604211 18789 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:58:59.604400 18789 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:58:59.604669 18789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:58:59.607081 18789 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } }
I20260807 02:58:59.607724 18789 leader_election.cc:302] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba162cb2f3b24ee79b23eefe6d90b62c; no voters:
I20260807 02:58:59.608479 18789 leader_election.cc:288] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:58:59.608575 18795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:58:59.609483 18795 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 1 LEADER]: Becoming Leader. State: Replica: ba162cb2f3b24ee79b23eefe6d90b62c, State: Running, Role: LEADER
I20260807 02:58:59.610109 18795 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } }
I20260807 02:58:59.611164 18789 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:58:59.612874 18797 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } } }
I20260807 02:58:59.613426 18797 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: This master's current role is: LEADER
I20260807 02:58:59.613051 18798 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba162cb2f3b24ee79b23eefe6d90b62c. Latest consensus state: current_term: 1 leader_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } } }
I20260807 02:58:59.613819 18798 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: This master's current role is: LEADER
I20260807 02:58:59.617923 18801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:58:59.623466 18801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:58:59.626241 18789 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 02:58:59.643380 18801 catalog_manager.cc:1347] Generated new cluster ID: b8021bcba9a64e05bd171a7fb15f6cdd
I20260807 02:58:59.643589 18801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:58:59.662299 18801 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:58:59.663906 18801 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:58:59.675731 18801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Generated new TSK 0
I20260807 02:58:59.676741 18801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:59:00.514424 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:42229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41153
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:59:00 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786071540, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 02:59:00.978768 18849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:00.979249 18849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:00.979437 18849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:00.992592 18849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:00.992770 18849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:00.993012 18849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:59:01.011111 18849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.121.84:42229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:01.013309 18849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:01.015328 18849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:01.033315 18866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:01.034365 18867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:01.042935 18849 server_base.cc:1034] running on GCE node
W20260807 02:59:01.043866 18870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:01.044792 18849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:01.046202 18849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:01.047524 18849 hybrid_clock.cc:648] HybridClock initialized: now 1786071541047454 us; error 40 us; skew 500 ppm
I20260807 02:59:01.048027 18849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:01 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071541, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:01.052822 18849 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:59:01.055233 18849 webserver.cc:460] Webserver started at http://127.1.121.65:33547/ using document root <none> and password file <none>
I20260807 02:59:01.056365 18849 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:01.056548 18849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:01.057009 18849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:59:01 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071541, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:59:01.084362 18849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "18be30ac745a4426ad5fdc77c0cb685b"
format_stamp: "Formatted at 2026-08-07 02:59:01 on dist-test-slave-3h5h"
server_key: "2bf8b8a85cfdf176f8564f85b3769fb1"
server_key_iv: "212bada8dfff3b66989f61a5f560742f"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:01.086438 18849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "18be30ac745a4426ad5fdc77c0cb685b"
format_stamp: "Formatted at 2026-08-07 02:59:01 on dist-test-slave-3h5h"
server_key: "2bf8b8a85cfdf176f8564f85b3769fb1"
server_key_iv: "212bada8dfff3b66989f61a5f560742f"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:01.095145 18849 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260807 02:59:01.125406 18884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:01.127501 18849 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260807 02:59:01.127765 18849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "18be30ac745a4426ad5fdc77c0cb685b"
format_stamp: "Formatted at 2026-08-07 02:59:01 on dist-test-slave-3h5h"
server_key: "2bf8b8a85cfdf176f8564f85b3769fb1"
server_key_iv: "212bada8dfff3b66989f61a5f560742f"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:01.128037 18849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:01.165756 18849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:01.171461 18849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:01.171866 18849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:01.173211 18849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:01.175953 18849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:01.176050 18849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:01.176246 18849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:01.176321 18849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:01.274322 18849 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:36739
I20260807 02:59:01.274360 19030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:36739 every 8 connection(s)
I20260807 02:59:01.276073 18849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:59:01.283329 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 18849
I20260807 02:59:01.283589 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:59:01.284463 1509 external_mini_cluster.cc:1428] Setting key 01d2928276d7db5cd27c65af995cb59b
Aug 07 02:59:01 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071541, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:01.342736 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:01.280715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41153 (local address 127.1.121.65:52431)
0807 02:59:01.315410 (+ 34695us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:01.315510 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:01.318591 (+ 3081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:01.319886 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:01.319900 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:01.320642 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:01.321651 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:01.321682 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:01.323447 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:01.323455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:01.325847 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:01.325865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:01.327387 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:01.327397 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:01.327756 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:01.328678 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:01.328715 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:01.332132 (+ 3417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:01.337211 (+ 5079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:01.337223 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:01.337244 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:01.337934 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:01.338474 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:01.338480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:01.338484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:01.338800 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:01.339632 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:01.339640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:01.340132 (+ 492us) client_negotiation.cc:769] Sending connection context
0807 02:59:01.340463 (+ 331us) client_negotiation.cc:240] Negotiation successful
0807 02:59:01.341094 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34241,"thread_start_us":58221,"threads_started":1}
I20260807 02:59:01.344266 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:01.282458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:52431 (local address 127.1.121.126:41153)
0807 02:59:01.314631 (+ 32173us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:01.314639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:01.319189 (+ 4550us) server_negotiation.cc:407] Connection header received
0807 02:59:01.319283 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:01.319287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:01.319408 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:01.319537 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:01.321912 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:01.323213 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:01.326070 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:01.326820 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:01.327008 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:01.332485 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:01.332496 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:01.332514 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:01.332572 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:01.337011 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:01.338123 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:01.338129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:01.338133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:01.338318 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:01.338965 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:01.338971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:01.338974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:01.339381 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:01.339555 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:01.343389 (+ 3834us) server_negotiation.cc:299] Negotiation successful
0807 02:59:01.343695 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32005,"thread_start_us":31796,"threads_started":1}
I20260807 02:59:01.346982 19031 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41153
I20260807 02:59:01.347277 19031 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:01.348273 19031 heartbeater.cc:511] Master 127.1.121.126:41153 requested a full tablet report, sending...
I20260807 02:59:01.351353 18739 ts_manager.cc:194] Registered new tserver with Master: 18be30ac745a4426ad5fdc77c0cb685b (127.1.121.65:36739)
I20260807 02:59:01.353752 18739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:52431
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Aug 07 02:59:01 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071540, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
W20260807 02:59:01.416671 1509 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.121.66: real 0.129s user 0.003s sys 0.026s
I20260807 02:59:01.417920 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:42229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41153
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:59:01 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071540, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
W20260807 02:59:01.961014 19040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:01.961488 19040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:01.961655 19040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:01.973495 19040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:01.973660 19040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:01.973945 19040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:59:01.990247 19040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.121.84:42229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:01.992451 19040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:01.994383 19040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:02.012266 19056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:02.012364 19060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:02.012364 19052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:02.013729 19040 server_base.cc:1034] running on GCE node
I20260807 02:59:02.014488 19040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:02.015779 19040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:02.017062 19040 hybrid_clock.cc:648] HybridClock initialized: now 1786071542016998 us; error 62 us; skew 500 ppm
I20260807 02:59:02.017455 19040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:02 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071542, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:02.021926 19040 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:59:02.023803 19040 webserver.cc:460] Webserver started at http://127.1.121.66:43167/ using document root <none> and password file <none>
I20260807 02:59:02.024823 19040 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:02.025012 19040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:02.025427 19040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:59:02 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071542, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:59:02.044480 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "a78a1e7c0bf9445a9bf92b0be3333b46"
format_stamp: "Formatted at 2026-08-07 02:59:02 on dist-test-slave-3h5h"
server_key: "a24acccab63105b5c6a67979ee1e404a"
server_key_iv: "22543842c3fc3034fb61208deb5a9c28"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:02.045522 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "a78a1e7c0bf9445a9bf92b0be3333b46"
format_stamp: "Formatted at 2026-08-07 02:59:02 on dist-test-slave-3h5h"
server_key: "a24acccab63105b5c6a67979ee1e404a"
server_key_iv: "22543842c3fc3034fb61208deb5a9c28"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:02.054170 19040 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260807 02:59:02.073174 19072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:02.075659 19040 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260807 02:59:02.075873 19040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "a78a1e7c0bf9445a9bf92b0be3333b46"
format_stamp: "Formatted at 2026-08-07 02:59:02 on dist-test-slave-3h5h"
server_key: "a24acccab63105b5c6a67979ee1e404a"
server_key_iv: "22543842c3fc3034fb61208deb5a9c28"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:02.076170 19040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:02.093447 19040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:02.098702 19040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:02.099126 19040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:02.100355 19040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:02.102581 19040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:02.102684 19040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:02.102818 19040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:02.102885 19040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:02.148070 19040 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:46565
I20260807 02:59:02.148089 19223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:46565 every 8 connection(s)
I20260807 02:59:02.149796 19040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:59:02.154575 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 19040
I20260807 02:59:02.154767 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:59:02.155481 1509 external_mini_cluster.cc:1428] Setting key 8860e6e09c1b2f9fec8c5353c4346a60
Aug 07 02:59:02 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071542, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:02.177024 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:02.152989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:47213 (local address 127.1.121.126:41153)
0807 02:59:02.153517 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:02.153522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:02.156279 (+ 2757us) server_negotiation.cc:407] Connection header received
0807 02:59:02.157494 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:02.157500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:02.157615 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:02.157741 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:02.159796 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.161039 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.163496 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.164264 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.164423 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:02.169136 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:02.169143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:02.169148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:02.169194 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:02.172887 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:02.173970 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:02.173976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:02.173980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:02.174149 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:02.174724 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:02.174729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:02.174732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:02.175123 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:02.175272 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:02.176314 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0807 02:59:02.176577 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":295,"threads_started":1}
I20260807 02:59:02.178638 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:02.153715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41153 (local address 127.1.121.66:47213)
0807 02:59:02.154722 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:02.154780 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:02.157140 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:02.158053 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:02.158065 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:02.158697 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:02.159589 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:02.159617 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.161258 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.161263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:02.163314 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:02.163344 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.164541 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.164549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:02.164901 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:02.165769 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:02.165802 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:02.168812 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:02.173064 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:02.173077 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:02.173100 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:02.173824 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:02.174283 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:02.174291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:02.174298 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:02.174587 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:02.175274 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:02.175285 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:02.175802 (+ 517us) client_negotiation.cc:769] Sending connection context
0807 02:59:02.176210 (+ 408us) client_negotiation.cc:240] Negotiation successful
0807 02:59:02.176822 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":411,"threads_started":1}
I20260807 02:59:02.180401 19224 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41153
I20260807 02:59:02.180807 19224 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:02.181802 19224 heartbeater.cc:511] Master 127.1.121.126:41153 requested a full tablet report, sending...
I20260807 02:59:02.183722 18739 ts_manager.cc:194] Registered new tserver with Master: a78a1e7c0bf9445a9bf92b0be3333b46 (127.1.121.66:46565)
I20260807 02:59:02.185070 18739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:47213
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:59:02.231997 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:42229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41153
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:59:02.365769 19031 heartbeater.cc:503] Master 127.1.121.126:41153 was elected leader, sending a full tablet report...
W20260807 02:59:02.598085 19234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:02.598428 19234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:02.598606 19234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:02.605904 19234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:02.606076 19234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:02.606251 19234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:59:02.617223 19234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.121.84:42229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:02.618683 19234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:02.620108 19234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:02.633894 19241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:02.633936 19245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:02.633975 19242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:02.635248 19234 server_base.cc:1034] running on GCE node
I20260807 02:59:02.635877 19234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:02.636940 19234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:02.638187 19234 hybrid_clock.cc:648] HybridClock initialized: now 1786071542638141 us; error 55 us; skew 500 ppm
I20260807 02:59:02.638526 19234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:02 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071542, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:02.641765 19234 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:59:02.643256 19234 webserver.cc:460] Webserver started at http://127.1.121.67:44145/ using document root <none> and password file <none>
I20260807 02:59:02.643909 19234 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:02.644012 19234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:02.644275 19234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:59:02 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071542, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:59:02.658591 19234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "e854dafff8be486080644c71d150106b"
format_stamp: "Formatted at 2026-08-07 02:59:02 on dist-test-slave-3h5h"
server_key: "0f7bc36e84344bab228ed52e0a836f48"
server_key_iv: "09b3027a71f43c6be658305719d875e9"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:02.659582 19234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "e854dafff8be486080644c71d150106b"
format_stamp: "Formatted at 2026-08-07 02:59:02 on dist-test-slave-3h5h"
server_key: "0f7bc36e84344bab228ed52e0a836f48"
server_key_iv: "09b3027a71f43c6be658305719d875e9"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:02.665757 19234 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260807 02:59:02.680655 19252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:02.682489 19234 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 02:59:02.682739 19234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "e854dafff8be486080644c71d150106b"
format_stamp: "Formatted at 2026-08-07 02:59:02 on dist-test-slave-3h5h"
server_key: "0f7bc36e84344bab228ed52e0a836f48"
server_key_iv: "09b3027a71f43c6be658305719d875e9"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:02.682976 19234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:02.703003 19234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:02.707396 19234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:02.707846 19234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:02.709097 19234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:02.710739 19234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:02.710850 19234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:02.711112 19234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:02.711242 19234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:02.773325 19234 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:38447
I20260807 02:59:02.773345 19417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:38447 every 8 connection(s)
I20260807 02:59:02.775116 19234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:59:02.784008 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 19234
I20260807 02:59:02.784235 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:59:02.784822 1509 external_mini_cluster.cc:1428] Setting key 2551e944ae1e618108a4ff0420a94562
Aug 07 02:59:02 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071542, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:02.800951 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:02.778454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:49047 (local address 127.1.121.126:41153)
0807 02:59:02.779067 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:02.779072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:02.780361 (+ 1289us) server_negotiation.cc:407] Connection header received
0807 02:59:02.782473 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:02.782479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:02.782571 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:02.782736 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:02.784291 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.785192 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.787560 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.789022 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.789162 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:02.793258 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:02.793265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:02.793271 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:02.793332 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:02.797143 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:02.797922 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:02.797928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:02.797932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:02.798106 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:02.798618 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:02.798624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:02.798629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:02.798926 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:02.799113 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:02.800042 (+ 929us) server_negotiation.cc:299] Negotiation successful
0807 02:59:02.800354 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":332,"threads_started":1}
I20260807 02:59:02.801959 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:02.779154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41153 (local address 127.1.121.67:49047)
0807 02:59:02.780158 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:02.780200 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:02.782183 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:02.782998 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:02.783007 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:02.783462 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:02.784070 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:02.784088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.785341 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.785348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:02.787395 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:02.787411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.789357 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.789363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:02.789597 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:02.790398 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:02.790420 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:02.792988 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:02.797297 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:02.797304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:02.797318 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:02.797793 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:02.798231 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:02.798234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:02.798237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:02.798488 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:02.799149 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:02.799158 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:02.799605 (+ 447us) client_negotiation.cc:769] Sending connection context
0807 02:59:02.799925 (+ 320us) client_negotiation.cc:240] Negotiation successful
0807 02:59:02.800499 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":427,"threads_started":1}
I20260807 02:59:02.803437 19418 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41153
I20260807 02:59:02.803843 19418 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:02.804766 19418 heartbeater.cc:511] Master 127.1.121.126:41153 requested a full tablet report, sending...
I20260807 02:59:02.806254 18739 ts_manager.cc:194] Registered new tserver with Master: e854dafff8be486080644c71d150106b (127.1.121.67:38447)
I20260807 02:59:02.807070 18739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:49047
I20260807 02:59:02.813715 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:59:02.815505 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 18849
Aug 07 02:59:02 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:59:02.834201 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:02.816741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.1.121.65:36739)
0807 02:59:02.817720 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:02.817728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:02.817753 (+ 25us) server_negotiation.cc:407] Connection header received
0807 02:59:02.817813 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:02.817820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:02.818038 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:02.818169 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:02.819833 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.821152 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.822753 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.823289 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.823386 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:02.827954 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:02.827966 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:02.827981 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:02.828017 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:02.831359 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:02.832024 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:02.832031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:02.832033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:02.832213 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:02.832546 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:02.832549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:02.832551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:02.832940 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:02.833246 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:02.833490 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 02:59:02.833756 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":491,"threads_started":1}
I20260807 02:59:02.835608 19025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:02.921627 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 18849
I20260807 02:59:02.945231 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 19040
Aug 07 02:59:02 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:59:02.967684 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:02.950941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.1.121.66:46565)
0807 02:59:02.951592 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:02.951606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:02.951634 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:59:02.951706 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:02.951713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:02.951935 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:02.952079 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:02.953487 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.954760 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.956154 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:02.956761 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:02.957040 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:02.961854 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:02.961867 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:02.961884 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:02.961921 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:02.964914 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:02.965693 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:02.965701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:02.965703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:02.965840 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:02.966167 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:02.966170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:02.966172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:02.966511 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:02.966769 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:02.967046 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 02:59:02.967258 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":368,"threads_started":1}
I20260807 02:59:02.969201 19219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:03.054217 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 19040
I20260807 02:59:03.079744 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 19234
Aug 07 02:59:03 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:59:03.098045 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:03.080772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.1.121.67:38447)
0807 02:59:03.081565 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:03.081573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:03.081603 (+ 30us) server_negotiation.cc:407] Connection header received
0807 02:59:03.081667 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:03.081673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:03.081896 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:03.082072 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:03.083537 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:03.084548 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:03.086080 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:03.086654 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:03.086819 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:03.091623 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:03.091634 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:03.091656 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:03.091690 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:03.094953 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:03.095745 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:03.095752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:03.095755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:03.095916 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:03.096348 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:03.096353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:03.096355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:03.096710 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:03.097082 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:03.097386 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 02:59:03.097679 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":475,"threads_started":1}
I20260807 02:59:03.099370 19410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:03.182767 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 19234
I20260807 02:59:03.206840 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 18690
I20260807 02:59:03.208249 18782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:03.322340 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 18690
I20260807 02:59:03.353272 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41153
--webserver_interface=127.1.121.126
--webserver_port=41657
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:41153
--ranger_config_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:42229/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.121.84:42229/kms with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:03.700891 19437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:03.701239 19437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:03.701413 19437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:03.709345 19437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:59:03.709543 19437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:03.709676 19437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:03.709798 19437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:59:03.709904 19437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:59:03.721692 19437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.121.84:42229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:41153
--ranger_config_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:03.723129 19437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:03.724567 19437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:03.737799 19449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:03.737828 19444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:03.737924 19443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:03.739112 19437 server_base.cc:1034] running on GCE node
I20260807 02:59:03.740072 19437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:03.741739 19437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:03.743059 19437 hybrid_clock.cc:648] HybridClock initialized: now 1786071543743006 us; error 45 us; skew 500 ppm
I20260807 02:59:03.743466 19437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:03 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071543, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:03.750754 19437 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:59:03.752403 19437 webserver.cc:460] Webserver started at http://127.1.121.126:41657/ using document root <none> and password file <none>
I20260807 02:59:03.753211 19437 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:03.753364 19437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 02:59:03 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071543, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:59:03.844667 19437 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:59:03.848081 19459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:03.849709 19437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:59:03.849916 19437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "ba162cb2f3b24ee79b23eefe6d90b62c"
format_stamp: "Formatted at 2026-08-07 02:58:59 on dist-test-slave-3h5h"
server_key: "a3b27b1a226e07f5acecc6d9bd698008"
server_key_iv: "ae4af33fadf0151c195ae2a7a4aee4cc"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:03.850895 19437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:03.885839 19437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:03.890375 19437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:03.890709 19437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:03.911537 19437 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:41153
I20260807 02:59:03.911542 19530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:41153 every 8 connection(s)
I20260807 02:59:03.913502 19437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:59:03.920656 19531 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:59:03.923683 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 19437
I20260807 02:59:03.924060 19531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Bootstrap starting.
I20260807 02:59:03.924628 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:42229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:36739
--local_ip_for_outbound_sockets=127.1.121.65
--tserver_master_addrs=127.1.121.126:41153
--webserver_port=33547
--webserver_interface=127.1.121.65
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.121.84:42229/kms with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:59:03.945151 19531 log.cc:824] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:03.955132 19531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:59:03.955789 19531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Bootstrap complete.
I20260807 02:59:03.965384 19531 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } }
I20260807 02:59:03.966437 19531 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba162cb2f3b24ee79b23eefe6d90b62c, State: Initialized, Role: FOLLOWER
I20260807 02:59:03.967160 19531 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } }
I20260807 02:59:03.967435 19531 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:59:03.967571 19531 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:59:03.967785 19531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:59:03.971750 19531 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } }
I20260807 02:59:03.972275 19531 leader_election.cc:302] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba162cb2f3b24ee79b23eefe6d90b62c; no voters:
I20260807 02:59:03.973021 19531 leader_election.cc:288] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 02:59:03.973173 19539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 2 FOLLOWER]: Leader election won for term 2
I20260807 02:59:03.974512 19539 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [term 2 LEADER]: Becoming Leader. State: Replica: ba162cb2f3b24ee79b23eefe6d90b62c, State: Running, Role: LEADER
I20260807 02:59:03.975153 19539 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } }
I20260807 02:59:03.975723 19531 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:59:03.977967 19541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba162cb2f3b24ee79b23eefe6d90b62c. Latest consensus state: current_term: 2 leader_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } } }
I20260807 02:59:03.978466 19541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:03.979116 19540 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba162cb2f3b24ee79b23eefe6d90b62c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41153 } } }
I20260807 02:59:03.979461 19540 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:03.983141 19548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:59:03.994336 19548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:59:03.996335 19548 catalog_manager.cc:1259] Loaded cluster ID: b8021bcba9a64e05bd171a7fb15f6cdd
I20260807 02:59:03.996555 19548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:59:04.001838 19548 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:59:04.003882 19548 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Loaded TSK: 0
I20260807 02:59:04.006001 19548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:59:04.013948 19531 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260807 02:59:04.419996 19533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:04.420590 19533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:04.420819 19533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:04.432665 19533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:04.432960 19533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:04.433195 19533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:59:04.449343 19533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.121.84:42229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:36739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=33547
--tserver_master_addrs=127.1.121.126:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:04.451521 19533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:04.453527 19533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:04.496447 19587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:04.510912 19584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:04.511525 19533 server_base.cc:1034] running on GCE node
W20260807 02:59:04.512907 19589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:04.513908 19533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:04.515533 19533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:04.516927 19533 hybrid_clock.cc:648] HybridClock initialized: now 1786071544516818 us; error 95 us; skew 500 ppm
I20260807 02:59:04.517345 19533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:04 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071544, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:04.521745 19533 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:59:04.524103 19533 webserver.cc:460] Webserver started at http://127.1.121.65:33547/ using document root <none> and password file <none>
I20260807 02:59:04.525120 19533 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:04.525317 19533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 02:59:04 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071544, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:59:04.575392 19533 fs_manager.cc:709] Time spent opening directory manager: real 0.027s user 0.015s sys 0.011s
I20260807 02:59:04.580921 19598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:04.582533 19533 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260807 02:59:04.582749 19533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "18be30ac745a4426ad5fdc77c0cb685b"
format_stamp: "Formatted at 2026-08-07 02:59:01 on dist-test-slave-3h5h"
server_key: "2bf8b8a85cfdf176f8564f85b3769fb1"
server_key_iv: "212bada8dfff3b66989f61a5f560742f"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:04.583791 19533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:04.603067 19533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:04.607600 19533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:04.607913 19533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:04.609074 19533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:04.617791 19533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:04.617920 19533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:04.618089 19533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:04.618191 19533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:04.707772 19533 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:36739
I20260807 02:59:04.707834 19761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:36739 every 8 connection(s)
I20260807 02:59:04.709435 19533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:59:04.718853 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 19533
I20260807 02:59:04.719568 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:42229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:46565
--local_ip_for_outbound_sockets=127.1.121.66
--tserver_master_addrs=127.1.121.126:41153
--webserver_port=43167
--webserver_interface=127.1.121.66
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.121.84:42229/kms with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:59:04 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071544, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:04.751546 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:04.713012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41153 (local address 127.1.121.65:35309)
0807 02:59:04.717280 (+ 4268us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:04.717326 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:04.724262 (+ 6936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:04.726993 (+ 2731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:04.727006 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:04.727689 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:04.728532 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:04.728562 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:04.731026 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:04.731035 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:04.733283 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:04.733302 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:04.735318 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:04.735327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:04.735644 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:04.736568 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:04.736601 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:04.739806 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:04.745185 (+ 5379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:04.745199 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:04.745220 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:04.745926 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:04.746635 (+ 709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:04.746641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:04.746645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:04.746973 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:04.748336 (+ 1363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:04.748345 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:04.748886 (+ 541us) client_negotiation.cc:769] Sending connection context
0807 02:59:04.749279 (+ 393us) client_negotiation.cc:240] Negotiation successful
0807 02:59:04.749975 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4015,"thread_start_us":3807,"threads_started":1}
I20260807 02:59:04.751565 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:04.712723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:35309 (local address 127.1.121.126:41153)
0807 02:59:04.713530 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:04.713544 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:04.717485 (+ 3941us) server_negotiation.cc:407] Connection header received
0807 02:59:04.725144 (+ 7659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:04.725171 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:04.725708 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:04.726128 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:04.728781 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:04.730802 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:04.733460 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:04.734636 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:04.734833 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:04.740166 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:04.740179 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:04.740201 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:04.740263 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:04.744983 (+ 4720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:04.746159 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:04.746171 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:04.746182 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:04.746427 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:04.747158 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:04.747165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:04.747169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:04.747981 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:04.748189 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:04.749592 (+ 1403us) server_negotiation.cc:299] Negotiation successful
0807 02:59:04.750091 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":291,"threads_started":1}
I20260807 02:59:04.756577 19763 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41153
I20260807 02:59:04.756987 19763 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:04.757988 19763 heartbeater.cc:511] Master 127.1.121.126:41153 requested a full tablet report, sending...
W20260807 02:59:04.762167 19763 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:41153 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 02:59:04.762423 19763 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:04.762759 19763 heartbeater.cc:511] Master 127.1.121.126:41153 requested a full tablet report, sending...
I20260807 02:59:04.764277 19763 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:04.764617 19763 heartbeater.cc:511] Master 127.1.121.126:41153 requested a full tablet report, sending...
W20260807 02:59:04.765862 19763 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 02:59:05.195421 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:05.195765 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:05.195935 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:05.203595 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:05.203748 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:05.203933 19771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:59:05.215446 19771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.121.84:42229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:46565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=43167
--tserver_master_addrs=127.1.121.126:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:05.216931 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:05.218286 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:05.237848 19791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:05.237914 19790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:05.239709 19795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:05.240180 19771 server_base.cc:1034] running on GCE node
I20260807 02:59:05.240897 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:05.241991 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:05.243223 19771 hybrid_clock.cc:648] HybridClock initialized: now 1786071545243155 us; error 63 us; skew 500 ppm
I20260807 02:59:05.243706 19771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:05 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071545, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:05.248339 19771 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:59:05.254006 19771 webserver.cc:460] Webserver started at http://127.1.121.66:43167/ using document root <none> and password file <none>
I20260807 02:59:05.254851 19771 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:05.255028 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 02:59:05 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071545, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:59:05.303776 19771 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.005s sys 0.012s
I20260807 02:59:05.308014 19805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:05.309845 19771 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:59:05.310060 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "a78a1e7c0bf9445a9bf92b0be3333b46"
format_stamp: "Formatted at 2026-08-07 02:59:02 on dist-test-slave-3h5h"
server_key: "a24acccab63105b5c6a67979ee1e404a"
server_key_iv: "22543842c3fc3034fb61208deb5a9c28"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:05.311379 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:05.360160 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:05.364728 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:05.365120 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:05.366413 19771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:05.368234 19771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:05.368371 19771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:05.368537 19771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:05.368673 19771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:59:05 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786071545, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:05.429759 19771 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:46565
I20260807 02:59:05.429821 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:46565 every 8 connection(s)
I20260807 02:59:05.432070 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:59:05.439689 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 19771
I20260807 02:59:05.440514 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.121.84:42229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:38447
--local_ip_for_outbound_sockets=127.1.121.67
--tserver_master_addrs=127.1.121.126:41153
--webserver_port=44145
--webserver_interface=127.1.121.67
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.121.84:42229/kms with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:59:05 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071545, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:05.475353 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:05.442906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41153 (local address 127.1.121.66:49949)
0807 02:59:05.449559 (+ 6653us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:05.449627 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:05.452537 (+ 2910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:05.453973 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:05.453986 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:05.454717 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:05.455693 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:05.455724 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:05.457439 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:05.457446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:05.459625 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:05.459642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:05.460962 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:05.460974 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:05.461280 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:05.462184 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:05.462218 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:05.465212 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:05.470113 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:05.470124 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:05.470143 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:05.470773 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:05.471268 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:05.471273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:05.471277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:05.471556 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:05.472323 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:05.472332 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:05.472853 (+ 521us) client_negotiation.cc:769] Sending connection context
0807 02:59:05.473208 (+ 355us) client_negotiation.cc:240] Negotiation successful
0807 02:59:05.473847 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6238,"thread_start_us":5963,"threads_started":1}
I20260807 02:59:05.476418 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:05.443670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:49949 (local address 127.1.121.126:41153)
0807 02:59:05.453318 (+ 9648us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:05.453324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:05.453342 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:59:05.453401 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:05.453404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:05.453471 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:05.453630 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:05.455945 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:05.457170 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:05.459840 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:05.460623 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:05.460787 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:05.465524 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:05.465532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:05.465537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:05.465582 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:05.469882 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:05.470935 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:05.470941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:05.470944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:05.471125 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:05.471701 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:05.471706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:05.471710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:05.472086 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:05.472230 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:05.475847 (+ 3617us) server_negotiation.cc:299] Negotiation successful
0807 02:59:05.476107 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9529,"thread_start_us":9419,"threads_started":1}
I20260807 02:59:05.478322 19982 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41153
I20260807 02:59:05.478679 19982 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:05.479470 19982 heartbeater.cc:511] Master 127.1.121.126:41153 requested a full tablet report, sending...
I20260807 02:59:05.481463 19483 ts_manager.cc:194] Registered new tserver with Master: a78a1e7c0bf9445a9bf92b0be3333b46 (127.1.121.66:46565)
I20260807 02:59:05.483060 19483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:49949
I20260807 02:59:05.766027 19763 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:05.766510 19763 heartbeater.cc:511] Master 127.1.121.126:41153 requested a full tablet report, sending...
I20260807 02:59:05.767886 19483 ts_manager.cc:194] Registered new tserver with Master: 18be30ac745a4426ad5fdc77c0cb685b (127.1.121.65:36739)
I20260807 02:59:05.769168 19483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:35309
Aug 07 02:59:05 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071545, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
W20260807 02:59:05.962909 19988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:05.963322 19988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:05.963526 19988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:05.974809 19988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:05.974970 19988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:05.975147 19988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:59:05.989182 19988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.121.84:42229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:38447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=44145
--tserver_master_addrs=127.1.121.126:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:05.991106 19988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:05.992622 19988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:06.011139 19997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:06.011292 19999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:06.011926 20002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:06.012938 19988 server_base.cc:1034] running on GCE node
I20260807 02:59:06.014002 19988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:06.015563 19988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:06.016896 19988 hybrid_clock.cc:648] HybridClock initialized: now 1786071546016804 us; error 77 us; skew 500 ppm
I20260807 02:59:06.017377 19988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:06 dist-test-slave-3h5h krb5kdc[18080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071546, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:06.023102 19988 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:59:06.028779 19988 webserver.cc:460] Webserver started at http://127.1.121.67:44145/ using document root <none> and password file <none>
I20260807 02:59:06.029726 19988 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:06.029911 19988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 02:59:06 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071546, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
I20260807 02:59:06.064054 19988 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 02:59:06.067586 20014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:06.069614 19988 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:59:06.069844 19988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "e854dafff8be486080644c71d150106b"
format_stamp: "Formatted at 2026-08-07 02:59:02 on dist-test-slave-3h5h"
server_key: "0f7bc36e84344bab228ed52e0a836f48"
server_key_iv: "09b3027a71f43c6be658305719d875e9"
server_key_version: "kuduclusterkey@0"
I20260807 02:59:06.071023 19988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:06.108565 19988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:06.113214 19988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:06.113617 19988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:06.114914 19988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:06.116528 19988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:06.116662 19988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:06.116803 19988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:06.116936 19988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 02:59:06.216547 19988 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:38447
I20260807 02:59:06.216645 20187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:38447 every 8 connection(s)
I20260807 02:59:06.218916 19988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:59:06.226475 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 19988
Aug 07 02:59:06 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071545, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for HTTP/127.1.121.84@KRBTEST.COM
Aug 07 02:59:06 dist-test-slave-3h5h krb5kdc[18080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071546, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:06.280807 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.227657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.1.121.126:41153)
0807 02:59:06.254489 (+ 26832us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.254499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.255432 (+ 933us) server_negotiation.cc:407] Connection header received
0807 02:59:06.255532 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.255538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.255623 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.255736 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:06.258351 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.259722 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.262249 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.263249 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.263352 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.270328 (+ 6976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:06.270335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:06.270340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:06.270391 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:06.274954 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:06.277421 (+ 2467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:06.277428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:06.277432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:06.277594 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:06.278075 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:06.278080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:06.278084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:06.278470 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:06.279315 (+ 845us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.279981 (+ 666us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.280338 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":26630,"thread_start_us":29524,"threads_started":1}
I20260807 02:59:06.287546 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.257414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:55971 (local address 127.1.121.126:41153)
0807 02:59:06.259394 (+ 1980us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.259401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.259424 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:59:06.259524 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.259530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.259635 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.260448 (+ 813us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:06.262466 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.265040 (+ 2574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.267813 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.268381 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.268531 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.274772 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:06.274781 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:06.274787 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:06.274849 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:06.281258 (+ 6409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:06.282605 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:06.282611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:06.282614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:06.282765 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:06.283388 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:06.283394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:06.283398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:06.283893 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:06.284040 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.286161 (+ 2121us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.286713 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1824,"thread_start_us":1599,"threads_started":1}
I20260807 02:59:06.289855 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.234680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41153 (local address 127.1.121.67:55971)
0807 02:59:06.249745 (+ 15065us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:06.249837 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:06.255656 (+ 5819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:06.260192 (+ 4536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:06.260209 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:06.261051 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:06.262150 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.262185 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.265309 (+ 3124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.265320 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.267640 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.267655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.268669 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.268678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.269071 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.270714 (+ 1643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:06.270752 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:06.274293 (+ 3541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:06.281409 (+ 7116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:06.281419 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:06.281440 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:06.282444 (+ 1004us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:06.282918 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:06.282923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:06.282927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:06.283240 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:06.284151 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:06.284161 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:06.284646 (+ 485us) client_negotiation.cc:769] Sending connection context
0807 02:59:06.285016 (+ 370us) client_negotiation.cc:240] Negotiation successful
0807 02:59:06.285625 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14546,"thread_start_us":17995,"threads_started":1}
I20260807 02:59:06.291718 20188 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41153
I20260807 02:59:06.292083 20188 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:06.293044 20188 heartbeater.cc:511] Master 127.1.121.126:41153 requested a full tablet report, sending...
I20260807 02:59:06.294824 19483 ts_manager.cc:194] Registered new tserver with Master: e854dafff8be486080644c71d150106b (127.1.121.67:38447)
I20260807 02:59:06.295979 19483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:55971
I20260807 02:59:06.297168 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:59:06.320544 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.300862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.1.121.126:41153)
0807 02:59:06.301037 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.301043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.301161 (+ 118us) server_negotiation.cc:407] Connection header received
0807 02:59:06.301453 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.301459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.301542 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.301658 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:06.303620 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.304353 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.306206 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.306884 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.306993 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.313183 (+ 6190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:06.313191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:06.313196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:06.313244 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:06.316919 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:06.317882 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:06.317889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:06.317893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:06.318112 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:06.318632 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:06.318637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:06.318641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:06.319026 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:06.319298 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.319941 (+ 643us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.320188 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 02:59:06.331264 19483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:59:06.342466 19483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:59:06.452792 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.387390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:36739 (local address 127.0.0.1:51370)
0807 02:59:06.432077 (+ 44687us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:06.432102 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:06.432250 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:06.434152 (+ 1902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:06.434156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:06.434178 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:06.434546 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.434559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.447955 (+ 13396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.447960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.449353 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.449366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.451754 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.451761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.451992 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.452057 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 02:59:06.452217 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 02:59:06.452437 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44517,"thread_start_us":39865,"threads_started":1}
I20260807 02:59:06.453832 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.388157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.1.121.65:36739)
0807 02:59:06.427031 (+ 38874us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.427042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.432406 (+ 5364us) server_negotiation.cc:407] Connection header received
0807 02:59:06.432542 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.432553 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.432882 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.433104 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:06.434809 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.437014 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.449559 (+ 12545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.451160 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.451307 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.451596 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.453267 (+ 1671us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.453465 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38649,"thread_start_us":38878,"threads_started":1}
I20260807 02:59:06.458519 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.376113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:38447 (local address 127.0.0.1:51074)
0807 02:59:06.425868 (+ 49755us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:06.425925 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:06.426178 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:06.451065 (+ 24887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:06.451071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:06.451103 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:06.451498 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.451510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.453487 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.453492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.455494 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.455510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.457737 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.457743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.457912 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.457961 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 02:59:06.458047 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 02:59:06.458180 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49398,"thread_start_us":52498,"threads_started":1}
I20260807 02:59:06.459311 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.377592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.1.121.67:38447)
0807 02:59:06.450062 (+ 72470us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.450073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.450113 (+ 40us) server_negotiation.cc:407] Connection header received
0807 02:59:06.450223 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.450232 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.450530 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.450768 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:06.451747 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.453337 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.455727 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.457154 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.457297 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.457554 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.458784 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.458956 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72268,"thread_start_us":72097,"threads_started":1}
I20260807 02:59:06.460213 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.377156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.1.121.66:46565)
0807 02:59:06.425111 (+ 47955us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.425124 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.430608 (+ 5484us) server_negotiation.cc:407] Connection header received
0807 02:59:06.430781 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.430793 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.431104 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.431358 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:06.437265 (+ 5907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.439399 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.450321 (+ 10922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.451891 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.452052 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.452283 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.459658 (+ 7375us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.459868 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47697,"thread_start_us":51040,"threads_started":1}
I20260807 02:59:06.459821 19664 tablet_service.cc:1467] Processing CreateTablet for tablet cfe35692e0f3476c8bc171662dd16270 (DEFAULT_TABLE table=test-table [id=701d29f6e2fc45a68f2da18892499028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:06.460902 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.376544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:46565 (local address 127.0.0.1:43358)
0807 02:59:06.428877 (+ 52333us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:06.428898 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:06.429020 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:06.433390 (+ 4370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:06.433399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:06.433430 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:06.433902 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.433921 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.439584 (+ 5663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.439589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.442588 (+ 2999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.442612 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.458999 (+ 16387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.459007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.459194 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.459228 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 02:59:06.459313 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 02:59:06.460531 (+ 1218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52180,"thread_start_us":73462,"threads_started":1}
I20260807 02:59:06.462517 19664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe35692e0f3476c8bc171662dd16270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:06.464757 19882 tablet_service.cc:1467] Processing CreateTablet for tablet cfe35692e0f3476c8bc171662dd16270 (DEFAULT_TABLE table=test-table [id=701d29f6e2fc45a68f2da18892499028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:06.464757 20082 tablet_service.cc:1467] Processing CreateTablet for tablet cfe35692e0f3476c8bc171662dd16270 (DEFAULT_TABLE table=test-table [id=701d29f6e2fc45a68f2da18892499028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:06.466238 20082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe35692e0f3476c8bc171662dd16270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:06.466241 19882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe35692e0f3476c8bc171662dd16270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:06.485060 20215 tablet_bootstrap.cc:492] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: Bootstrap starting.
I20260807 02:59:06.486753 19982 heartbeater.cc:503] Master 127.1.121.126:41153 was elected leader, sending a full tablet report...
I20260807 02:59:06.495435 20214 tablet_bootstrap.cc:492] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: Bootstrap starting.
I20260807 02:59:06.499526 20215 tablet_bootstrap.cc:654] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:06.501070 20215 log.cc:824] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:06.502000 20214 tablet_bootstrap.cc:654] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:06.503037 20216 tablet_bootstrap.cc:492] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: Bootstrap starting.
I20260807 02:59:06.503481 20214 log.cc:824] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:06.508124 20216 tablet_bootstrap.cc:654] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:06.509696 20216 log.cc:824] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:06.511897 20214 tablet_bootstrap.cc:492] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: No bootstrap required, opened a new log
I20260807 02:59:06.512337 20214 ts_tablet_manager.cc:1397] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.004s
I20260807 02:59:06.519797 20216 tablet_bootstrap.cc:492] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: No bootstrap required, opened a new log
I20260807 02:59:06.520147 20216 ts_tablet_manager.cc:1397] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260807 02:59:06.521338 20215 tablet_bootstrap.cc:492] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: No bootstrap required, opened a new log
I20260807 02:59:06.521668 20215 ts_tablet_manager.cc:1397] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.005s
I20260807 02:59:06.532091 20216 raft_consensus.cc:348] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } }
I20260807 02:59:06.532668 20216 raft_consensus.cc:374] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:06.532786 20216 raft_consensus.cc:729] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e854dafff8be486080644c71d150106b, State: Initialized, Role: FOLLOWER
I20260807 02:59:06.533736 20216 consensus_queue.cc:260] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } }
I20260807 02:59:06.540671 20214 raft_consensus.cc:348] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } }
I20260807 02:59:06.540817 20215 raft_consensus.cc:348] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } }
I20260807 02:59:06.541206 20214 raft_consensus.cc:374] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:06.541306 20214 raft_consensus.cc:729] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18be30ac745a4426ad5fdc77c0cb685b, State: Initialized, Role: FOLLOWER
I20260807 02:59:06.541326 20215 raft_consensus.cc:374] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:06.541429 20215 raft_consensus.cc:729] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78a1e7c0bf9445a9bf92b0be3333b46, State: Initialized, Role: FOLLOWER
I20260807 02:59:06.542009 20214 consensus_queue.cc:260] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } }
I20260807 02:59:06.542884 20216 ts_tablet_manager.cc:1428] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: Time spent starting tablet: real 0.023s user 0.011s sys 0.011s
I20260807 02:59:06.543318 20188 heartbeater.cc:503] Master 127.1.121.126:41153 was elected leader, sending a full tablet report...
I20260807 02:59:06.544598 20215 consensus_queue.cc:260] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } }
I20260807 02:59:06.551132 19763 heartbeater.cc:503] Master 127.1.121.126:41153 was elected leader, sending a full tablet report...
I20260807 02:59:06.551821 20214 ts_tablet_manager.cc:1428] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: Time spent starting tablet: real 0.039s user 0.022s sys 0.004s
I20260807 02:59:06.553553 20215 ts_tablet_manager.cc:1428] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: Time spent starting tablet: real 0.032s user 0.020s sys 0.004s
I20260807 02:59:06.690550 20222 raft_consensus.cc:482] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:06.690848 20222 raft_consensus.cc:504] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } }
I20260807 02:59:06.692521 20222 leader_election.cc:288] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a78a1e7c0bf9445a9bf92b0be3333b46 (127.1.121.66:46565), 18be30ac745a4426ad5fdc77c0cb685b (127.1.121.65:36739)
I20260807 02:59:06.700202 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.693128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:40457 (local address 127.1.121.65:36739)
0807 02:59:06.693356 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.693364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.693647 (+ 283us) server_negotiation.cc:407] Connection header received
0807 02:59:06.693859 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.693863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.693960 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.694065 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:06.694549 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.695661 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.698350 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.699131 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.699346 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.699422 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.699774 (+ 352us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.699925 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 02:59:06.700240 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.693133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:36739 (local address 127.1.121.67:40457)
0807 02:59:06.693479 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:06.693503 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:06.693671 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:06.694099 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:06.694102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:06.694122 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:06.694395 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.694406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.695811 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.695815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.698128 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.698146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.699337 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.699344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.699597 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.699624 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 02:59:06.699743 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 02:59:06.699972 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":117}
I20260807 02:59:06.700296 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.693201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:49455 (local address 127.1.121.66:46565)
0807 02:59:06.693443 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.693453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.693945 (+ 492us) server_negotiation.cc:407] Connection header received
0807 02:59:06.694028 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.694035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.694125 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.694220 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:06.694851 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.695900 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.698364 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.699132 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.699350 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.699424 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.699779 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.699927 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 02:59:06.700736 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.693169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:46565 (local address 127.1.121.67:49455)
0807 02:59:06.693837 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:06.693857 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:06.693941 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:06.694245 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:06.694249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:06.694267 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:06.694686 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.694695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.696052 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.696059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.698240 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:06.698254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.699356 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.699362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:06.699602 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.699627 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 02:59:06.699743 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 02:59:06.699933 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":109,"thread_start_us":347,"threads_started":1}
I20260807 02:59:06.701294 19695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe35692e0f3476c8bc171662dd16270" candidate_uuid: "e854dafff8be486080644c71d150106b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18be30ac745a4426ad5fdc77c0cb685b" is_pre_election: true
I20260807 02:59:06.701938 19695 raft_consensus.cc:2454] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e854dafff8be486080644c71d150106b in term 0.
I20260807 02:59:06.701963 19910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe35692e0f3476c8bc171662dd16270" candidate_uuid: "e854dafff8be486080644c71d150106b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" is_pre_election: true
I20260807 02:59:06.702507 19910 raft_consensus.cc:2454] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e854dafff8be486080644c71d150106b in term 0.
I20260807 02:59:06.702826 20019 leader_election.cc:302] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18be30ac745a4426ad5fdc77c0cb685b, e854dafff8be486080644c71d150106b; no voters:
I20260807 02:59:06.703299 20222 raft_consensus.cc:2781] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:59:06.703444 20222 raft_consensus.cc:482] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:59:06.703562 20222 raft_consensus.cc:3037] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:06.705888 20222 raft_consensus.cc:504] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } }
I20260807 02:59:06.706563 20222 leader_election.cc:288] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [CANDIDATE]: Term 1 election: Requested vote from peers a78a1e7c0bf9445a9bf92b0be3333b46 (127.1.121.66:46565), 18be30ac745a4426ad5fdc77c0cb685b (127.1.121.65:36739)
I20260807 02:59:06.706923 19910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe35692e0f3476c8bc171662dd16270" candidate_uuid: "e854dafff8be486080644c71d150106b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46"
I20260807 02:59:06.707161 19910 raft_consensus.cc:3037] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:06.707188 19695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe35692e0f3476c8bc171662dd16270" candidate_uuid: "e854dafff8be486080644c71d150106b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18be30ac745a4426ad5fdc77c0cb685b"
I20260807 02:59:06.707393 19695 raft_consensus.cc:3037] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [term 0 FOLLOWER]: Advancing to term 1
W20260807 02:59:06.709148 19983 tablet.cc:2369] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:06.710608 19910 raft_consensus.cc:2454] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e854dafff8be486080644c71d150106b in term 1.
I20260807 02:59:06.710757 19695 raft_consensus.cc:2454] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e854dafff8be486080644c71d150106b in term 1.
I20260807 02:59:06.711078 20018 leader_election.cc:302] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a78a1e7c0bf9445a9bf92b0be3333b46, e854dafff8be486080644c71d150106b; no voters:
I20260807 02:59:06.711403 20222 raft_consensus.cc:2781] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:59:06.711915 20222 raft_consensus.cc:686] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 1 LEADER]: Becoming Leader. State: Replica: e854dafff8be486080644c71d150106b, State: Running, Role: LEADER
I20260807 02:59:06.712471 20222 consensus_queue.cc:237] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } }
W20260807 02:59:06.713413 19766 tablet.cc:2369] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:06.724241 19483 catalog_manager.cc:5495] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b reported cstate change: term changed from 0 to 1, leader changed from <none> to e854dafff8be486080644c71d150106b (127.1.121.67). New cstate: current_term: 1 leader_uuid: "e854dafff8be486080644c71d150106b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e854dafff8be486080644c71d150106b" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 } health_report { overall_health: UNKNOWN } } }
W20260807 02:59:06.757516 20189 tablet.cc:2369] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:06.798269 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.793040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.1.121.67:38447)
0807 02:59:06.793252 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.793257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.793275 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:59:06.793436 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.793440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.793520 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.793657 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:59:06.794010 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.794737 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.796149 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.796529 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.796623 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.797148 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:59:06.797680 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:59:06.797788 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.797870 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.797955 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 02:59:06.806969 19695 raft_consensus.cc:1261] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [term 1 FOLLOWER]: Refusing update from remote peer e854dafff8be486080644c71d150106b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:06.807005 19910 raft_consensus.cc:1261] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [term 1 FOLLOWER]: Refusing update from remote peer e854dafff8be486080644c71d150106b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:06.808086 20237 consensus_queue.cc:1035] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18be30ac745a4426ad5fdc77c0cb685b" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 36739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:06.808561 20222 consensus_queue.cc:1035] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a78a1e7c0bf9445a9bf92b0be3333b46" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:06.835359 19482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:59:06.835667 19482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:59:06.839753 19482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba162cb2f3b24ee79b23eefe6d90b62c: Sending DeleteTablet for 3 replicas of tablet cfe35692e0f3476c8bc171662dd16270
I20260807 02:59:06.841215 20082 tablet_service.cc:1514] Processing DeleteTablet for tablet cfe35692e0f3476c8bc171662dd16270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:06 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:51074
I20260807 02:59:06.841507 19664 tablet_service.cc:1514] Processing DeleteTablet for tablet cfe35692e0f3476c8bc171662dd16270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:06 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:51370
I20260807 02:59:06.842094 19882 tablet_service.cc:1514] Processing DeleteTablet for tablet cfe35692e0f3476c8bc171662dd16270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:06 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:43358
I20260807 02:59:06.844511 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 19533
I20260807 02:59:06.847119 20260 tablet_replica.cc:330] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: stopping tablet replica
I20260807 02:59:06.847777 20260 raft_consensus.cc:2229] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:59:06.848798 20260 raft_consensus.cc:2258] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:06.850090 20262 tablet_replica.cc:330] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: stopping tablet replica
I20260807 02:59:06.850612 20262 raft_consensus.cc:2229] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:59:06.851209 20262 raft_consensus.cc:2258] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:06.852208 20263 tablet_replica.cc:330] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: stopping tablet replica
I20260807 02:59:06.852411 20260 ts_tablet_manager.cc:1905] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:06.852705 20263 raft_consensus.cc:2229] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:59:06.853312 20263 raft_consensus.cc:2258] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:06.856585 20263 ts_tablet_manager.cc:1905] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:06.857445 20262 ts_tablet_manager.cc:1905] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:06.861817 20260 ts_tablet_manager.cc:1918] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:06.862144 20260 log.cc:1196] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/cfe35692e0f3476c8bc171662dd16270
I20260807 02:59:06.862936 20260 ts_tablet_manager.cc:1939] T cfe35692e0f3476c8bc171662dd16270 P e854dafff8be486080644c71d150106b: Deleting consensus metadata
I20260807 02:59:06.865224 19460 catalog_manager.cc:4841] TS e854dafff8be486080644c71d150106b (127.1.121.67:38447): tablet cfe35692e0f3476c8bc171662dd16270 (table test-table [id=701d29f6e2fc45a68f2da18892499028]) successfully deleted
I20260807 02:59:06.865483 20262 ts_tablet_manager.cc:1918] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:06.865649 20262 log.cc:1196] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/cfe35692e0f3476c8bc171662dd16270
I20260807 02:59:06.867031 20262 ts_tablet_manager.cc:1939] T cfe35692e0f3476c8bc171662dd16270 P 18be30ac745a4426ad5fdc77c0cb685b: Deleting consensus metadata
I20260807 02:59:06.869035 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:06.845505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.1.121.65:36739)
0807 02:59:06.845730 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:06.845740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:06.850252 (+ 4512us) server_negotiation.cc:407] Connection header received
0807 02:59:06.850523 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:06.850532 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:06.850622 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:06.850723 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:06.852771 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.854006 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.856119 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:06.856792 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:06.856934 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:06.860767 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:06.860785 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:06.860803 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:06.860880 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:06.865112 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:06.866109 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:06.866118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:06.866121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:06.866308 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:06.866809 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:06.866815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:06.866818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:06.867317 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:06.867615 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:06.868262 (+ 647us) server_negotiation.cc:299] Negotiation successful
0807 02:59:06.868547 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:59:06.869164 19466 catalog_manager.cc:4841] TS 18be30ac745a4426ad5fdc77c0cb685b (127.1.121.65:36739): tablet cfe35692e0f3476c8bc171662dd16270 (table test-table [id=701d29f6e2fc45a68f2da18892499028]) successfully deleted
I20260807 02:59:06.870887 19755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:06.875192 20263 ts_tablet_manager.cc:1918] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:06.875495 20263 log.cc:1196] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/cfe35692e0f3476c8bc171662dd16270
I20260807 02:59:06.876197 20263 ts_tablet_manager.cc:1939] T cfe35692e0f3476c8bc171662dd16270 P a78a1e7c0bf9445a9bf92b0be3333b46: Deleting consensus metadata
I20260807 02:59:06.878332 19464 catalog_manager.cc:4841] TS a78a1e7c0bf9445a9bf92b0be3333b46 (127.1.121.66:46565): tablet cfe35692e0f3476c8bc171662dd16270 (table test-table [id=701d29f6e2fc45a68f2da18892499028]) successfully deleted
I20260807 02:59:06.995124 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 19533
I20260807 02:59:07.021677 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 19771
I20260807 02:59:07.036957 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:07.022556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.1.121.66:46565)
0807 02:59:07.022772 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:07.022779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:07.022803 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:59:07.022943 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:07.022949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:07.023038 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:07.023157 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:07.024460 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:07.025308 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:07.026647 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:07.027065 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:07.027211 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:07.030149 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:07.030165 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:07.030186 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:07.030227 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:07.033588 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:07.034367 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:07.034375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:07.034378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:07.034532 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:07.035029 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:07.035036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:07.035040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:07.035593 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:07.035958 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:07.036394 (+ 436us) server_negotiation.cc:299] Negotiation successful
0807 02:59:07.036576 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 02:59:07.037731 19976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:07.130079 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 19771
I20260807 02:59:07.156574 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 19988
I20260807 02:59:07.174793 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:07.157548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.1.121.67:38447)
0807 02:59:07.157753 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:07.157758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:07.157889 (+ 131us) server_negotiation.cc:407] Connection header received
0807 02:59:07.158099 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:07.158103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:07.158169 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:07.158266 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:07.160413 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:07.161473 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:07.163620 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:07.164056 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:07.164154 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:07.168035 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:07.168048 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:07.168065 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:07.168106 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:07.171444 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:07.172447 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:07.172458 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:07.172461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:07.172622 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:07.173126 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:07.173129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:07.173131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:07.173528 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:07.173790 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:07.174250 (+ 460us) server_negotiation.cc:299] Negotiation successful
0807 02:59:07.174433 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 02:59:07.175815 20182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:07.313203 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 19988
I20260807 02:59:07.341305 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 19437
I20260807 02:59:07.342531 19523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:07.458001 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 19437
I20260807 02:59:07.489550 1509 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 02:59:07.835364 1509 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 02:59:07.835613 1509 mini_ranger.cc:66] Stopping Ranger...
I20260807 02:59:08.192124 1509 mini_ranger.cc:68] Stopped Ranger
2026-08-07 02:59:08.192 UTC [18119] LOG: received smart shutdown request
2026-08-07 02:59:08.195 UTC [18119] LOG: background worker "logical replication launcher" (PID 18132) exited with exit code 1
2026-08-07 02:59:08.196 UTC [18125] LOG: shutting down
2026-08-07 02:59:08.371 UTC [18119] LOG: database system is shut down
2026-08-07T02:59:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56683 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:59:08 dist-test-slave-3h5h krb5kdc[20281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:59:08 dist-test-slave-3h5h krb5kdc[20281](info): set up 2 sockets
Aug 07 02:59:08 dist-test-slave-3h5h krb5kdc[20281](info): commencing operation
krb5kdc: starting...
W20260807 02:59:09.845618 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.288s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:59:10 dist-test-slave-3h5h krb5kdc[20281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:59:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:59:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:59:10.157497 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:39471
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:39471,127.1.121.125:38407,127.1.121.124:40343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:10.528695 20300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:10.529109 20300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:10.529277 20300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:10.536710 20300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:59:10.536847 20300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:10.536994 20300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:10.537096 20300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:59:10.537161 20300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:59:10.547483 20300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:39471,127.1.121.125:38407,127.1.121.124:40343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:39471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:10.549041 20300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:10.550469 20300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:10.562886 20307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:10.562947 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:10.563089 20306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:10.563760 20300 server_base.cc:1034] running on GCE node
I20260807 02:59:10.564522 20300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:10.566007 20300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:10.567216 20300 hybrid_clock.cc:648] HybridClock initialized: now 1786071550567157 us; error 52 us; skew 500 ppm
I20260807 02:59:10.567531 20300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:10 dist-test-slave-3h5h krb5kdc[20281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:10.574123 20300 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:59:10.575639 20300 webserver.cc:460] Webserver started at http://127.1.121.126:43231/ using document root <none> and password file <none>
I20260807 02:59:10.576383 20300 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:10.576498 20300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:10.576774 20300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:10.579962 20300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "59295680ea6e4a51827c64d30770eb5c"
format_stamp: "Formatted at 2026-08-07 02:59:10 on dist-test-slave-3h5h"
server_key: "42bc3bd91c8dda5cd3e877ca121a4909"
server_key_iv: "ab1a82de3df53559ef12ec824973426e"
server_key_version: "encryptionkey@0"
I20260807 02:59:10.580758 20300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "59295680ea6e4a51827c64d30770eb5c"
format_stamp: "Formatted at 2026-08-07 02:59:10 on dist-test-slave-3h5h"
server_key: "42bc3bd91c8dda5cd3e877ca121a4909"
server_key_iv: "ab1a82de3df53559ef12ec824973426e"
server_key_version: "encryptionkey@0"
I20260807 02:59:10.586784 20300 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 02:59:10.590912 20323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:10.592383 20300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:59:10.592602 20300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "59295680ea6e4a51827c64d30770eb5c"
format_stamp: "Formatted at 2026-08-07 02:59:10 on dist-test-slave-3h5h"
server_key: "42bc3bd91c8dda5cd3e877ca121a4909"
server_key_iv: "ab1a82de3df53559ef12ec824973426e"
server_key_version: "encryptionkey@0"
I20260807 02:59:10.592828 20300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:10.626372 20300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:10.631170 20300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:10.631496 20300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:10.651378 20300 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:39471
I20260807 02:59:10.651412 20403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:39471 every 8 connection(s)
I20260807 02:59:10.652889 20300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:59:10.657130 20404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:10.658794 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 20300
I20260807 02:59:10.659037 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:59:10.659739 1509 external_mini_cluster.cc:1428] Setting key 689611f336a7f076f9c25de038306323
I20260807 02:59:10.665132 20404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:59:10 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.121.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.125@KRBTEST.COM" created.
I20260807 02:59:10.693208 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.668807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.1.121.126:39471)
0807 02:59:10.670914 (+ 2107us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:10.670931 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:10.671004 (+ 73us) server_negotiation.cc:407] Connection header received
0807 02:59:10.672796 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:10.672824 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:10.673155 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:10.673402 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:10.674998 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:10.676456 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:10.678188 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:10.678839 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:10.679104 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:10.683296 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:10.683310 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:10.683330 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:10.683385 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:10.687543 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:10.688633 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:10.688642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:10.688645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:10.688830 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:10.689469 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:10.689475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:10.689479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:10.690363 (+ 884us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:10.690621 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:10.691212 (+ 591us) server_negotiation.cc:299] Negotiation successful
0807 02:59:10.691625 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":1804,"spinlock_wait_cycles":17664,"thread_start_us":1533,"threads_started":1}
I20260807 02:59:10.693233 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.668806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:39471 (local address 127.0.0.1:35630)
0807 02:59:10.669950 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:10.670013 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:10.672359 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:10.673429 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:10.673448 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:10.674340 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:10.674748 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:10.674769 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:10.676658 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:10.676670 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:10.678008 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:10.678022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:10.679138 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:10.679147 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:10.679431 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:10.680252 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:10.680281 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:10.682978 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:10.687741 (+ 4763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:10.687754 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:10.687766 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:10.688426 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:10.689143 (+ 717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:10.689167 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:10.689169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:10.689328 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:10.690570 (+ 1242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:10.690578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:10.690801 (+ 223us) client_negotiation.cc:769] Sending connection context
0807 02:59:10.691057 (+ 256us) client_negotiation.cc:240] Negotiation successful
0807 02:59:10.691748 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":204,"thread_start_us":532,"threads_started":1}
I20260807 02:59:10.696678 20404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:59:10.697939 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.697342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56438)
0807 02:59:10.697587 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:10.697747 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260807 02:59:10.698215 20326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
I20260807 02:59:10.699993 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.699500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56444)
0807 02:59:10.699696 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:10.699801 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 02:59:10.700224 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:38407: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab.
Entry for principal kudu/127.1.121.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab.
I20260807 02:59:10.728446 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.125:38407
--webserver_interface=127.1.121.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab
--principal=kudu/127.1.121.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:39471,127.1.121.125:38407,127.1.121.124:40343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:59:10.749755 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } attempt: 1
I20260807 02:59:10.751135 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.750645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56458)
0807 02:59:10.750870 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:10.750994 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 02:59:10.752105 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.751766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56470)
0807 02:59:10.751914 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:10.751986 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 02:59:10.752295 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:38407: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
I20260807 02:59:10.820580 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } attempt: 2
I20260807 02:59:10.821942 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.821400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56486)
0807 02:59:10.821683 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:10.821776 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 02:59:10.823005 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.822628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56502)
0807 02:59:10.822806 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:10.822871 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 02:59:10.823203 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:38407: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
I20260807 02:59:10.914484 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } attempt: 3
I20260807 02:59:10.915721 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.915224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56512)
0807 02:59:10.915447 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:10.915562 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 02:59:10.916743 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:10.916376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56526)
0807 02:59:10.916532 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:10.916605 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 02:59:10.916998 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:38407: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
I20260807 02:59:11.060251 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } attempt: 4
I20260807 02:59:11.061631 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.061025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56528)
0807 02:59:11.061396 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.061490 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260807 02:59:11.062744 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.062348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56540)
0807 02:59:11.062512 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.062611 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 02:59:11.062942 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:38407: Network error: Client connection negotiation failed: client connection to 127.1.121.125:38407: connect: Connection refused (error 111)
W20260807 02:59:11.086392 20412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:11.086755 20412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:11.086921 20412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:11.094774 20412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:59:11.095078 20412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:11.095220 20412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:11.095331 20412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:59:11.095409 20412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:59:11.106016 20412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:39471,127.1.121.125:38407,127.1.121.124:40343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab
--principal=kudu/127.1.121.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.125:38407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.121.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:11.107614 20412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:11.109117 20412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:11.121534 20421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:11.121609 20418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:11.121618 20419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:11.122404 20412 server_base.cc:1034] running on GCE node
I20260807 02:59:11.123231 20412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:11.124660 20412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:11.125900 20412 hybrid_clock.cc:648] HybridClock initialized: now 1786071551125860 us; error 51 us; skew 500 ppm
I20260807 02:59:11.126262 20412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:11.129529 20412 init.cc:377] Logged in from keytab as kudu/127.1.121.125@KRBTEST.COM (short username kudu)
I20260807 02:59:11.131080 20412 webserver.cc:460] Webserver started at http://127.1.121.125:33959/ using document root <none> and password file <none>
I20260807 02:59:11.131816 20412 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:11.131970 20412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:11.132290 20412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:11.135414 20412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/data/instance:
uuid: "e1265c8045ec404f86c475028497bd6d"
format_stamp: "Formatted at 2026-08-07 02:59:11 on dist-test-slave-3h5h"
server_key: "fc9347196af7fb3f665c8a44a99a47b3"
server_key_iv: "4ab655a4b5bc774ff4c486b2564529a8"
server_key_version: "encryptionkey@0"
I20260807 02:59:11.136257 20412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/wal/instance:
uuid: "e1265c8045ec404f86c475028497bd6d"
format_stamp: "Formatted at 2026-08-07 02:59:11 on dist-test-slave-3h5h"
server_key: "fc9347196af7fb3f665c8a44a99a47b3"
server_key_iv: "4ab655a4b5bc774ff4c486b2564529a8"
server_key_version: "encryptionkey@0"
I20260807 02:59:11.141661 20412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260807 02:59:11.145255 20428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:11.146636 20412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 02:59:11.146847 20412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/wal
uuid: "e1265c8045ec404f86c475028497bd6d"
format_stamp: "Formatted at 2026-08-07 02:59:11 on dist-test-slave-3h5h"
server_key: "fc9347196af7fb3f665c8a44a99a47b3"
server_key_iv: "4ab655a4b5bc774ff4c486b2564529a8"
server_key_version: "encryptionkey@0"
I20260807 02:59:11.147054 20412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:11.160794 20412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:11.165525 20412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:11.165823 20412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:11.184231 20412 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.125:38407
I20260807 02:59:11.184264 20504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.125:38407 every 8 connection(s)
I20260807 02:59:11.185691 20412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
I20260807 02:59:11.189252 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 20412
I20260807 02:59:11.189461 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-1/wal/instance
I20260807 02:59:11.189569 20506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:11.190002 1509 external_mini_cluster.cc:1428] Setting key d6b96d3340ddd1154c76a06e83b06d99
I20260807 02:59:11.196357 20506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.121.124@KRBTEST.COM; defaulting to no policy
I20260807 02:59:11.217991 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.199172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.1.121.126:39471)
0807 02:59:11.199668 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.199673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.200740 (+ 1067us) server_negotiation.cc:407] Connection header received
0807 02:59:11.203388 (+ 2648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.203395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.203492 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.203614 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.205223 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.205929 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.207603 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.208022 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.208167 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.211891 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.211896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.211900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.211944 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.214772 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.215543 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.215548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.215550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.215676 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.216112 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.216116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.216118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.216394 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.216568 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.217381 (+ 813us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.217602 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":280,"threads_started":1}
I20260807 02:59:11.218957 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.199801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:39471 (local address 127.0.0.1:35644)
0807 02:59:11.200571 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.200615 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:11.203106 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:11.203870 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:11.203879 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:11.204394 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:11.205042 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.205062 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.206103 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.206107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.207458 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.207470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.208265 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.208272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.208489 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.209236 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:11.209258 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:11.211649 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:11.214925 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.214935 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.214953 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.215400 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.215792 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.215797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.215801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.215980 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.216531 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:11.216539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:11.216997 (+ 458us) client_negotiation.cc:769] Sending connection context
0807 02:59:11.217272 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 02:59:11.217780 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.124@KRBTEST.COM" created.
I20260807 02:59:11.221122 20506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 02:59:11.238512 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.222163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.1.121.125:38407)
0807 02:59:11.222871 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.222881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.222916 (+ 35us) server_negotiation.cc:407] Connection header received
0807 02:59:11.222980 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.222990 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.223218 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.223390 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.224197 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.226038 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.227218 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.228239 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.228404 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.231741 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.231759 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.231783 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.231830 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.235195 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.236091 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.236098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.236100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.236229 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.236681 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.236685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.236687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.237173 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.237349 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.237811 (+ 462us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.238136 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":514,"thread_start_us":351,"threads_started":1}
I20260807 02:59:11.238538 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.221879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56550)
0807 02:59:11.222104 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.222130 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:11.222250 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:11.223401 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:11.223405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:11.223880 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:11.224020 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.224029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.226212 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.226216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.227062 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.227073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.228390 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.228395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.228563 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.229218 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:11.229235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:11.231492 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:11.235336 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.235343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.235350 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.235942 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.236349 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.236356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.236363 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.236561 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.237318 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:11.237326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:11.237493 (+ 167us) client_negotiation.cc:769] Sending connection context
0807 02:59:11.237661 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 02:59:11.237991 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 02:59:11.241401 20506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:59:11.242704 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.242245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57516)
0807 02:59:11.242431 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.242543 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 02:59:11.242990 20429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
I20260807 02:59:11.244634 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.244295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57520)
0807 02:59:11.244456 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.244535 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 02:59:11.244889 20506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:40343: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab.
Entry for principal kudu/127.1.121.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab.
I20260807 02:59:11.259759 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.124:40343
--webserver_interface=127.1.121.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab
--principal=kudu/127.1.121.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:39471,127.1.121.125:38407,127.1.121.124:40343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:59:11.294217 20506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } attempt: 1
I20260807 02:59:11.295506 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.294965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57522)
0807 02:59:11.295228 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.295343 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 02:59:11.296926 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.296396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57526)
0807 02:59:11.296655 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.296748 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260807 02:59:11.297266 20506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:40343: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
I20260807 02:59:11.362191 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } attempt: 5
I20260807 02:59:11.365497 20506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } attempt: 2
I20260807 02:59:11.366591 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.366052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57540)
0807 02:59:11.366262 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.366386 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 02:59:11.367836 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.367333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57556)
0807 02:59:11.367565 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.367654 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260807 02:59:11.368095 20506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:40343: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 02:59:11.377054 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.363053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.1.121.125:38407)
0807 02:59:11.363268 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.363276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.363376 (+ 100us) server_negotiation.cc:407] Connection header received
0807 02:59:11.363628 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.363632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.363717 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.363832 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.364664 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.365432 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.366955 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.367632 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.367762 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.371622 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.371627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.371631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.371664 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.374368 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.375045 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.375052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.375055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.375184 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.375573 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.375578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.375581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.375884 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.376046 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.376467 (+ 421us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.376702 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 02:59:11.377051 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.362975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56566)
0807 02:59:11.363259 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.363281 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:11.363463 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:11.363870 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:11.363875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:11.364307 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:11.364500 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.364509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.365578 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.365586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.366768 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.366785 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.367833 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.367841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.368141 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.368885 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:11.368916 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:11.371413 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:11.374521 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.374526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.374531 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.374912 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.375306 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.375311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.375315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.375456 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.376013 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:11.376018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:11.376196 (+ 178us) client_negotiation.cc:769] Sending connection context
0807 02:59:11.376356 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 02:59:11.376675 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260807 02:59:11.378465 20404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:59:11.379585 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.379194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57570)
0807 02:59:11.379369 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.379451 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 02:59:11.380585 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.380229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57576)
0807 02:59:11.380397 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.380461 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 02:59:11.380872 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:40343: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
I20260807 02:59:11.432120 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } attempt: 1
I20260807 02:59:11.433584 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.432884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57586)
0807 02:59:11.433193 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.433332 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 02:59:11.434851 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.434392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57590)
0807 02:59:11.434584 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.434693 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 02:59:11.435120 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:40343: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
I20260807 02:59:11.459326 20506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } attempt: 3
I20260807 02:59:11.460649 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.459984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57600)
0807 02:59:11.460267 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.460417 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 02:59:11.461953 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.461433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57602)
0807 02:59:11.461690 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.461792 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260807 02:59:11.462208 20506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:40343: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
I20260807 02:59:11.503386 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } attempt: 2
I20260807 02:59:11.504806 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.504160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57608)
0807 02:59:11.504451 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.504577 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 02:59:11.505970 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.505637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57620)
0807 02:59:11.505785 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.505862 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 02:59:11.506269 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:40343: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
I20260807 02:59:11.605450 20506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } attempt: 4
I20260807 02:59:11.606788 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.606154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57634)
0807 02:59:11.606444 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.606551 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 02:59:11.608255 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.607751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57644)
0807 02:59:11.607966 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.608085 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 02:59:11.608566 20506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:40343: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
I20260807 02:59:11.612527 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } attempt: 3
I20260807 02:59:11.613710 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.613247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57656)
0807 02:59:11.613448 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.613531 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260807 02:59:11.614771 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.614414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57662)
0807 02:59:11.614588 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.614666 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260807 02:59:11.615034 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:40343: Network error: Client connection negotiation failed: client connection to 127.1.121.124:40343: connect: Connection refused (error 111)
W20260807 02:59:11.616113 20514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:11.616412 20514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:11.616588 20514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:11.623930 20514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:59:11.624032 20514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:11.624135 20514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:11.624224 20514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:59:11.624290 20514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:59:11.634333 20514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:39471,127.1.121.125:38407,127.1.121.124:40343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab
--principal=kudu/127.1.121.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.124:40343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.121.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:11.635656 20514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:11.637042 20514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:11.649432 20520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:11.649482 20521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:11.649390 20523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:11.658361 20514 server_base.cc:1034] running on GCE node
I20260807 02:59:11.659262 20514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:11.660763 20514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:11.662038 20514 hybrid_clock.cc:648] HybridClock initialized: now 1786071551661987 us; error 43 us; skew 500 ppm
I20260807 02:59:11.662420 20514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:11.665673 20514 init.cc:377] Logged in from keytab as kudu/127.1.121.124@KRBTEST.COM (short username kudu)
I20260807 02:59:11.667250 20514 webserver.cc:460] Webserver started at http://127.1.121.124:43559/ using document root <none> and password file <none>
I20260807 02:59:11.668002 20514 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:11.668159 20514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:11.668514 20514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:11.671590 20514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/data/instance:
uuid: "ce9f41cb1bd24807b5156ef99caf44ff"
format_stamp: "Formatted at 2026-08-07 02:59:11 on dist-test-slave-3h5h"
server_key: "e76847ec62344cddf21e35232809e82d"
server_key_iv: "e23aee77f3e04afbc9e265317721624c"
server_key_version: "encryptionkey@0"
I20260807 02:59:11.672417 20514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/wal/instance:
uuid: "ce9f41cb1bd24807b5156ef99caf44ff"
format_stamp: "Formatted at 2026-08-07 02:59:11 on dist-test-slave-3h5h"
server_key: "e76847ec62344cddf21e35232809e82d"
server_key_iv: "e23aee77f3e04afbc9e265317721624c"
server_key_version: "encryptionkey@0"
I20260807 02:59:11.677809 20514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 02:59:11.681380 20535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:11.682785 20514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:59:11.682969 20514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/wal
uuid: "ce9f41cb1bd24807b5156ef99caf44ff"
format_stamp: "Formatted at 2026-08-07 02:59:11 on dist-test-slave-3h5h"
server_key: "e76847ec62344cddf21e35232809e82d"
server_key_iv: "e23aee77f3e04afbc9e265317721624c"
server_key_version: "encryptionkey@0"
I20260807 02:59:11.683202 20514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:11.703765 20514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:11.707899 20514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:11.708199 20514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:11.727234 20514 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.124:40343
I20260807 02:59:11.727268 20612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.124:40343 every 8 connection(s)
I20260807 02:59:11.728672 20514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
I20260807 02:59:11.732610 20613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:11.738305 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 20514
I20260807 02:59:11.738519 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786071323612974-1509-0/minicluster-data/master-2/wal/instance
I20260807 02:59:11.738989 1509 external_mini_cluster.cc:1428] Setting key cd426dc6481e66f7d8341f090223c207
I20260807 02:59:11.739449 20613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
I20260807 02:59:11.760473 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.742792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.1.121.126:39471)
0807 02:59:11.743364 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.743370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.743944 (+ 574us) server_negotiation.cc:407] Connection header received
0807 02:59:11.744721 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.744725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.744787 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.744931 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.746372 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.747133 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.749014 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.749645 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.749785 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.753514 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.753520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.753523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.753567 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.756971 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.757796 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.757802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.757805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.757968 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.758380 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.758385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.758389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.758811 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.758952 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.759874 (+ 922us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.760059 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":244,"threads_started":1}
I20260807 02:59:11.761518 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.742782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:39471 (local address 127.0.0.1:35652)
0807 02:59:11.743770 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.743827 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:11.744464 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:11.745020 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:11.745027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:11.745791 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:11.746190 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.746207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.747305 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.747311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.748800 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.748813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.749913 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.749921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.750173 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.751013 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:11.751035 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:11.753295 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:11.757137 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.757148 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.757159 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.757665 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.758082 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.758086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.758088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.758259 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.758961 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:11.758972 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:11.759429 (+ 457us) client_negotiation.cc:769] Sending connection context
0807 02:59:11.759738 (+ 309us) client_negotiation.cc:240] Negotiation successful
0807 02:59:11.760270 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":508,"threads_started":1}
I20260807 02:59:11.762171 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.741042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.1.121.124:40343)
0807 02:59:11.741915 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.741929 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.741975 (+ 46us) server_negotiation.cc:407] Connection header received
0807 02:59:11.744464 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.744493 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.744796 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.745034 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.746482 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.748033 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.749781 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.750592 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.750692 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.755111 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.755122 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.755145 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.755188 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.758724 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.759580 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.759586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.759588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.759764 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.760232 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.760239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.760244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.760695 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.761050 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.761480 (+ 430us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.761742 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":369,"threads_started":1}
I20260807 02:59:11.763307 20613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:11.776818 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.763941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:38407 (local address 127.0.0.1:56574)
0807 02:59:11.764194 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.764214 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:11.764304 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:11.764699 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:11.764704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:11.765142 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:11.765349 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.765360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.766373 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.766381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.767261 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.767272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.767988 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.767994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.768202 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.768770 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:11.768786 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:11.770758 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:11.774365 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.774371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.774375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.774761 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.775216 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.775219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.775222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.775356 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.775856 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:11.775861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:11.776013 (+ 152us) client_negotiation.cc:769] Sending connection context
0807 02:59:11.776168 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 02:59:11.776445 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260807 02:59:11.776866 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.764069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.1.121.125:38407)
0807 02:59:11.764285 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.764292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.764316 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:59:11.764443 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.764448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.764538 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.764632 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.765543 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.766231 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.767407 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.767833 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.767966 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.770962 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.770972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.770981 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.771028 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.774227 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.774895 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.774906 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.774908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.775087 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.775461 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.775465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.775467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.775726 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.775895 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.776250 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.776466 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:59:11.777835 20613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:59:11.780499 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.765363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.1.121.126:39471)
0807 02:59:11.765569 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.765575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.765591 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:59:11.765649 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.765652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.765713 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.765816 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.767069 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.767738 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.769083 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.769498 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.769636 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.774509 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.774515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.774518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.774551 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.777685 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.778432 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.778440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.778445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.778617 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.779072 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.779076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.779078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.779358 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.779654 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.779957 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.780210 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 02:59:11.791774 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.778765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.1.121.124:40343)
0807 02:59:11.778917 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.778921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.778978 (+ 57us) server_negotiation.cc:407] Connection header received
0807 02:59:11.779146 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.779151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.779224 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.779321 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.780005 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.781023 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.782447 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.783071 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.783167 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.786472 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.786476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.786480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.786508 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.789249 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.789848 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.789854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.789858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.789973 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.790336 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.790343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.790346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.790630 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.790785 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.791166 (+ 381us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.791405 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 02:59:11.791860 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.778615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57676)
0807 02:59:11.778911 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.778929 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:11.779046 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:11.779341 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:11.779345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:11.779743 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:11.779869 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.779878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.781187 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.781191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.782288 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.782303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.783223 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.783232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.783468 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.784037 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:11.784062 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:11.786288 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:11.789377 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.789385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.789388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.789727 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.790083 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.790089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.790093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.790230 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.790756 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:11.790760 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:11.790905 (+ 145us) client_negotiation.cc:769] Sending connection context
0807 02:59:11.791081 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 02:59:11.791519 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":139}
I20260807 02:59:11.792263 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } attempt: 4
I20260807 02:59:11.797711 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.782374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.1.121.125:38407)
0807 02:59:11.782545 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.782551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.782566 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:59:11.782709 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.782717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.782789 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.782889 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.784498 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.785270 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.786644 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.787247 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.787394 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.791735 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.791740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.791743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.791779 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.794852 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.795505 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.795512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.795516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.795667 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.796047 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.796052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.796055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.796426 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.796708 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.797115 (+ 407us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.797378 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:59:11.798909 20613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff: Bootstrap starting.
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
I20260807 02:59:11.803452 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:59:11.803457 20613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:11.804633 20613 log.cc:824] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:11.807979 20613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff: No bootstrap required, opened a new log
I20260807 02:59:11.810243 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.792890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57678)
0807 02:59:11.793154 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.793171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:11.793331 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:11.793703 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:11.793708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:11.794204 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:11.794433 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.794447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.796056 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.796062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.797634 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.797650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.798830 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.798837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.799034 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.799657 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:11.799689 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:11.802525 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:11.806723 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.806729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.806733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.807296 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.807794 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.807802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.807808 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.808027 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.809100 (+ 1073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:11.809105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:11.809323 (+ 218us) client_negotiation.cc:769] Sending connection context
0807 02:59:11.809485 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 02:59:11.809845 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260807 02:59:11.811406 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.792904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.1.121.124:40343)
0807 02:59:11.793095 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.793101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.793265 (+ 164us) server_negotiation.cc:407] Connection header received
0807 02:59:11.793451 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.793456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.793519 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.794582 (+ 1063us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.794740 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.795865 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.797979 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.798686 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.798809 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.802746 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.802752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.802757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.802793 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.806553 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.807457 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.807463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.807466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.807632 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.808212 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.808220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.808226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.808785 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.809027 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.810714 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.810963 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:59:11.818790 20404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c: Bootstrap starting.
I20260807 02:59:11.821714 20613 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } }
I20260807 02:59:11.822444 20613 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:11.822655 20613 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9f41cb1bd24807b5156ef99caf44ff, State: Initialized, Role: FOLLOWER
I20260807 02:59:11.823491 20404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:11.823666 20613 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } }
I20260807 02:59:11.824906 20404 log.cc:824] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:11.834750 20404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c: No bootstrap required, opened a new log
I20260807 02:59:11.837895 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:11.839442 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.811708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.1.121.126:39471)
0807 02:59:11.811887 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.811893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.818810 (+ 6917us) server_negotiation.cc:407] Connection header received
0807 02:59:11.819332 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.819343 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.819420 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.819491 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.821978 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.823120 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.825669 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.826600 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.826723 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.831850 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.831863 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.831869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.831956 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.835761 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.836778 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.836784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.836788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.836986 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.837486 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.837493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.837501 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.837881 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.838197 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.838760 (+ 563us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.839010 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 02:59:11.840556 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:59:11.840420 20613 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:59:11.843542 20637 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:11.843918 20637 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } }
I20260807 02:59:11.844511 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.811539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.1.121.124:40343)
0807 02:59:11.819055 (+ 7516us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.819062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.824112 (+ 5050us) server_negotiation.cc:407] Connection header received
0807 02:59:11.825214 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.825224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.825328 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.825413 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.828469 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.829722 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.832202 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.833212 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.833356 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.837128 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.837135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.837139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.837182 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.841002 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.841930 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.841936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.841939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.842101 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.842567 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.842572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.842576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.843008 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.843288 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.843923 (+ 635us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.844162 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7393}
I20260807 02:59:11.845700 20637 leader_election.cc:288] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59295680ea6e4a51827c64d30770eb5c (127.1.121.126:39471), e1265c8045ec404f86c475028497bd6d (127.1.121.125:38407)
I20260807 02:59:11.848685 20404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } }
I20260807 02:59:11.849264 20404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:11.849462 20404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59295680ea6e4a51827c64d30770eb5c, State: Initialized, Role: FOLLOWER
I20260807 02:59:11.850411 20404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } }
I20260807 02:59:11.852429 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:11.853672 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:59:11.853899 20404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:59:11.853971 20370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59295680ea6e4a51827c64d30770eb5c" is_pre_election: true
I20260807 02:59:11.855049 20370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9f41cb1bd24807b5156ef99caf44ff in term 0.
I20260807 02:59:11.855858 20540 leader_election.cc:302] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59295680ea6e4a51827c64d30770eb5c, ce9f41cb1bd24807b5156ef99caf44ff; no voters:
I20260807 02:59:11.856386 20637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:59:11.856539 20637 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:59:11.856644 20637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:11.858705 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.824808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.1.121.125:38407)
0807 02:59:11.825016 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.825022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.828540 (+ 3518us) server_negotiation.cc:407] Connection header received
0807 02:59:11.829441 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.829447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.829528 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.829593 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.833824 (+ 4231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.834998 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.842200 (+ 7202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.842954 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.843074 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.849012 (+ 5938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.849021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.849028 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.849089 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.853265 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.854273 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.854283 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.854289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.854493 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.855041 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.855048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.855051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.855741 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.857437 (+ 1696us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.857661 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.858127 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 02:59:11.861073 20637 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } }
I20260807 02:59:11.861966 20637 leader_election.cc:288] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [CANDIDATE]: Term 1 election: Requested vote from peers 59295680ea6e4a51827c64d30770eb5c (127.1.121.126:39471), e1265c8045ec404f86c475028497bd6d (127.1.121.125:38407)
I20260807 02:59:11.862584 20370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59295680ea6e4a51827c64d30770eb5c"
I20260807 02:59:11.862840 20370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:11.866556 20370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9f41cb1bd24807b5156ef99caf44ff in term 1.
I20260807 02:59:11.866995 20540 leader_election.cc:302] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59295680ea6e4a51827c64d30770eb5c, ce9f41cb1bd24807b5156ef99caf44ff; no voters:
I20260807 02:59:11.867328 20637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:59:11.868028 20637 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [term 1 LEADER]: Becoming Leader. State: Replica: ce9f41cb1bd24807b5156ef99caf44ff, State: Running, Role: LEADER
I20260807 02:59:11.868667 20637 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } }
I20260807 02:59:11.877357 20665 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce9f41cb1bd24807b5156ef99caf44ff. Latest consensus state: current_term: 1 leader_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:11.877838 20665 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [sys.catalog]: This master's current role is: LEADER
W20260807 02:59:11.887066 20333 tablet.cc:2369] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:59:11.888849 20683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:59:11.888973 20683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:59:11.889101 20681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:59:11.891812 20681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:59:11.907800 20506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } attempt: 5
W20260807 02:59:11.911818 20686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:59:11.912024 20686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:59:11.901672 20300 master_runner.cc:186] Error getting master registration for 127.1.121.125:38407: OK, instance_id { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" instance_seqno: 1786071551178999 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 02:59:11.915978 20300 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 02:59:11.883754 20514 master_runner.cc:186] Error getting master registration for 127.1.121.126:39471: OK, instance_id { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" instance_seqno: 1786071550645657 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 02:59:11.918779 20514 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 07 02:59:11 dist-test-slave-3h5h krb5kdc[20281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
I20260807 02:59:11.930003 20370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [term 1 FOLLOWER]: Refusing update from remote peer ce9f41cb1bd24807b5156ef99caf44ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:11.931066 20665 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:11.935331 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.908563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:40343 (local address 127.0.0.1:57688)
0807 02:59:11.908788 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:11.908813 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:11.908967 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:11.913496 (+ 4529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:11.913502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:11.914120 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:11.914351 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.914365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.916217 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.916227 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.917885 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:11.917904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.919122 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.919129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:11.919390 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.920124 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:11.920154 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:11.923641 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:11.930713 (+ 7072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.930719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.930724 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.931354 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.931871 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:11.931877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:11.931880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:11.932193 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:11.933103 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:11.933110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:11.933458 (+ 348us) client_negotiation.cc:769] Sending connection context
0807 02:59:11.933781 (+ 323us) client_negotiation.cc:240] Negotiation successful
0807 02:59:11.934456 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 02:59:11.935883 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:11.912735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.1.121.124:40343)
0807 02:59:11.912972 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:11.912979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:11.913001 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:59:11.913064 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:11.913070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:11.913159 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:11.913276 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:11.914554 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.916009 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.918081 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:11.918935 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:11.919083 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:11.923899 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:11.923908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:11.923914 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:11.923965 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:11.929009 (+ 5044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.931523 (+ 2514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.931529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.931533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.931705 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:11.932369 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:11.932378 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:11.932385 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:11.932803 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:11.933026 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:11.934130 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0807 02:59:11.934767 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:59:11.943279 20506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d: Bootstrap starting.
I20260807 02:59:11.949297 20506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:11.951715 20506 log.cc:824] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d: Log is configured to *not* fsync() on all Append() calls
W20260807 02:59:11.960381 20544 tablet.cc:2369] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:11.962735 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce9f41cb1bd24807b5156ef99caf44ff. Latest consensus state: current_term: 1 leader_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:11.963079 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:59:11.963815 20506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d: No bootstrap required, opened a new log
I20260807 02:59:11.963840 20691 mvcc.cc:205] Tried to move back new op lower bound from 7315749076671434752 to 7315749076459565056. Current Snapshot: MvccSnapshot[applied={T|T < 7315749076671434752}]
I20260807 02:59:11.964797 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:11.965057 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:59:11.974825 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:11.975205 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:11.976190 20681 catalog_manager.cc:1347] Generated new cluster ID: 1c6f133d88fa4d2e9eb5b881ef29a322
I20260807 02:59:11.976308 20681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:59:11.981035 20692 mvcc.cc:205] Tried to move back new op lower bound from 7315749076671434752 to 7315749076459565056. Current Snapshot: MvccSnapshot[applied={T|T < 7315749076671434752 or (T in {7315749076671434752})}]
I20260807 02:59:12.001945 20506 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } }
I20260807 02:59:12.003295 20506 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:12.003604 20506 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1265c8045ec404f86c475028497bd6d, State: Initialized, Role: FOLLOWER
I20260807 02:59:12.005508 20506 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } }
I20260807 02:59:12.011523 20681 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:59:12.013773 20681 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:59:12.021512 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:12.022099 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:59:12.025849 20468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1265c8045ec404f86c475028497bd6d" is_pre_election: true
I20260807 02:59:12.026573 20468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9f41cb1bd24807b5156ef99caf44ff in term 0.
I20260807 02:59:12.027036 20467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1265c8045ec404f86c475028497bd6d"
I20260807 02:59:12.027313 20467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:12.031152 20467 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9f41cb1bd24807b5156ef99caf44ff in term 1.
I20260807 02:59:12.032088 20466 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [term 1 FOLLOWER]: Refusing update from remote peer ce9f41cb1bd24807b5156ef99caf44ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:12.034412 20637 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:12.036541 20681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff: Generated new TSK 0
I20260807 02:59:12.037773 20681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:59:12.042822 20506 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:59:12.084336 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce9f41cb1bd24807b5156ef99caf44ff. Latest consensus state: current_term: 1 leader_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:12.084786 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:59:12.084462 20708 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:12.085022 20708 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:59:12.127842 20726 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 02:59:12.442373 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59295680ea6e4a51827c64d30770eb5c" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 39471 } } peers { permanent_uuid: "e1265c8045ec404f86c475028497bd6d" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 38407 } } peers { permanent_uuid: "ce9f41cb1bd24807b5156ef99caf44ff" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 40343 } } }
I20260807 02:59:12.442775 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:12.914144 20686 catalog_manager.cc:1259] Loaded cluster ID: 1c6f133d88fa4d2e9eb5b881ef29a322
I20260807 02:59:12.914254 20686 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c: loading cluster ID for follower catalog manager: success
I20260807 02:59:12.916626 20686 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c: acquiring CA information for follower catalog manager: success
I20260807 02:59:12.918467 20686 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59295680ea6e4a51827c64d30770eb5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 02:59:13.132162 20726 catalog_manager.cc:1259] Loaded cluster ID: 1c6f133d88fa4d2e9eb5b881ef29a322
I20260807 02:59:13.132272 20726 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d: loading cluster ID for follower catalog manager: success
I20260807 02:59:13.134927 20726 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d: acquiring CA information for follower catalog manager: success
I20260807 02:59:13.136787 20726 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1265c8045ec404f86c475028497bd6d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 02:59:13.199506 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 20300
I20260807 02:59:13.200873 20398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:13.325448 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 20300
I20260807 02:59:13.356997 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 20412
I20260807 02:59:13.358291 20497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:13.398118 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:13.397269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:39471 (local address 127.0.0.1:35686)
0807 02:59:13.397805 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:13.397949 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:39471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":333,"threads_started":1}
W20260807 02:59:13.398447 20540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.126:39471: connect: Connection refused (error 111)
I20260807 02:59:13.401994 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:13.401616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:39471 (local address 127.0.0.1:35700)
0807 02:59:13.401777 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:13.401872 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:39471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 02:59:13.402199 20540 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce9f41cb1bd24807b5156ef99caf44ff -> Peer 59295680ea6e4a51827c64d30770eb5c (127.1.121.126:39471): Couldn't send request to peer 59295680ea6e4a51827c64d30770eb5c. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.126:39471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 02:59:13.470826 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 20412
I20260807 02:59:13.500258 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 20514
I20260807 02:59:13.502188 20608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:13.613037 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 20514
2026-08-07T02:59:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5107 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:59:13 dist-test-slave-3h5h krb5kdc[20739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:59:13 dist-test-slave-3h5h krb5kdc[20739](info): set up 2 sockets
Aug 07 02:59:13 dist-test-slave-3h5h krb5kdc[20739](info): commencing operation
krb5kdc: starting...
W20260807 02:59:15.021232 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.356s user 0.006s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:59:15 dist-test-slave-3h5h krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:59:15.333614 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41525
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:41525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:15.684804 20761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:15.685189 20761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:15.685362 20761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:15.692728 20761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:59:15.692909 20761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:15.693017 20761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:15.693122 20761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:59:15.693204 20761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:59:15.703358 20761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:41525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:15.704783 20761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:15.706154 20761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:15.718796 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:15.718757 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:15.718910 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:15.719559 20761 server_base.cc:1034] running on GCE node
I20260807 02:59:15.720434 20761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:15.722000 20761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:15.723256 20761 hybrid_clock.cc:648] HybridClock initialized: now 1786071555723210 us; error 75 us; skew 500 ppm
I20260807 02:59:15.723637 20761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:15 dist-test-slave-3h5h krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071555, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:15.731768 20761 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:59:15.733826 20761 webserver.cc:460] Webserver started at http://127.1.121.126:33609/ using document root <none> and password file <none>
I20260807 02:59:15.734629 20761 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:15.734786 20761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:15.735181 20761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:15.738543 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "2e666bb475ce4715b7851e3fa355831d"
format_stamp: "Formatted at 2026-08-07 02:59:15 on dist-test-slave-3h5h"
server_key: "3b8935b1229bf53cd7eeb993938032dd"
server_key_iv: "864e483603081b58b544741183123c9f"
server_key_version: "encryptionkey@0"
I20260807 02:59:15.739387 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "2e666bb475ce4715b7851e3fa355831d"
format_stamp: "Formatted at 2026-08-07 02:59:15 on dist-test-slave-3h5h"
server_key: "3b8935b1229bf53cd7eeb993938032dd"
server_key_iv: "864e483603081b58b544741183123c9f"
server_key_version: "encryptionkey@0"
I20260807 02:59:15.745016 20761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 02:59:15.748764 20779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:15.750296 20761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:59:15.750509 20761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "2e666bb475ce4715b7851e3fa355831d"
format_stamp: "Formatted at 2026-08-07 02:59:15 on dist-test-slave-3h5h"
server_key: "3b8935b1229bf53cd7eeb993938032dd"
server_key_iv: "864e483603081b58b544741183123c9f"
server_key_version: "encryptionkey@0"
I20260807 02:59:15.750738 20761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:15.769129 20761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:15.774628 20761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:15.774973 20761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:15.795450 20761 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:41525
I20260807 02:59:15.795455 20853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:41525 every 8 connection(s)
I20260807 02:59:15.797083 20761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:59:15.801163 20854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:15.804144 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 20761
I20260807 02:59:15.804383 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:59:15.805280 1509 external_mini_cluster.cc:1428] Setting key 11a31f9b08b1df16fdc493b9b9aa18f7
I20260807 02:59:15.814805 20854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d: Bootstrap starting.
I20260807 02:59:15.818876 20854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:15.820312 20854 log.cc:824] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:59:15 dist-test-slave-3h5h krb5kdc[20739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:15.823477 20854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d: No bootstrap required, opened a new log
I20260807 02:59:15.831234 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:15.807536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.1.121.126:41525)
0807 02:59:15.808520 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:15.808532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:15.808587 (+ 55us) server_negotiation.cc:407] Connection header received
0807 02:59:15.810543 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:15.810569 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:15.811027 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:15.811412 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:15.813429 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:15.814918 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:15.816943 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:15.817685 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:15.817839 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:15.823208 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:15.823217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:15.823233 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:15.823279 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:15.826634 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:15.827544 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:15.827551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:15.827558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:15.827747 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:15.828181 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:15.828185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:15.828187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:15.828759 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:15.829147 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:15.829666 (+ 519us) server_negotiation.cc:299] Negotiation successful
0807 02:59:15.830153 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":436,"threads_started":1}
I20260807 02:59:15.831799 20854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e666bb475ce4715b7851e3fa355831d" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41525 } }
I20260807 02:59:15.832196 20854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:15.832325 20854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e666bb475ce4715b7851e3fa355831d, State: Initialized, Role: FOLLOWER
I20260807 02:59:15.833046 20854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e666bb475ce4715b7851e3fa355831d" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41525 } }
I20260807 02:59:15.833382 20854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:59:15.833544 20854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:59:15.833757 20854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:15.835793 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:59:15.836480 20854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e666bb475ce4715b7851e3fa355831d" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41525 } }
I20260807 02:59:15.837070 20854 leader_election.cc:302] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e666bb475ce4715b7851e3fa355831d; no voters:
I20260807 02:59:15.837749 20854 leader_election.cc:288] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:59:15.837859 20863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:59:15.839558 20863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [term 1 LEADER]: Becoming Leader. State: Replica: 2e666bb475ce4715b7851e3fa355831d, State: Running, Role: LEADER
I20260807 02:59:15.840126 20863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e666bb475ce4715b7851e3fa355831d" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41525 } }
I20260807 02:59:15.840322 20854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:59:15.843463 20867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e666bb475ce4715b7851e3fa355831d. Latest consensus state: current_term: 1 leader_uuid: "2e666bb475ce4715b7851e3fa355831d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e666bb475ce4715b7851e3fa355831d" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41525 } } }
I20260807 02:59:15.843883 20867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:15.844441 20865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e666bb475ce4715b7851e3fa355831d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e666bb475ce4715b7851e3fa355831d" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41525 } } }
I20260807 02:59:15.844748 20865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:15.845932 20879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:59:15.852139 20879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:59:15.861123 20879 catalog_manager.cc:1347] Generated new cluster ID: 5cb0760a415e4b34abda07024872b18e
I20260807 02:59:15.861248 20879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:59:15.880674 20879 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:59:15.882087 20879 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:59:15.899731 20879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e666bb475ce4715b7851e3fa355831d: Generated new TSK 0
I20260807 02:59:15.900707 20879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:59:15.949994 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 20761
I20260807 02:59:15.951417 20846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:16.051353 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 20761
2026-08-07T02:59:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2432 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:59:16 dist-test-slave-3h5h krb5kdc[20902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:59:16 dist-test-slave-3h5h krb5kdc[20902](info): set up 2 sockets
Aug 07 02:59:16 dist-test-slave-3h5h krb5kdc[20902](info): commencing operation
krb5kdc: starting...
W20260807 02:59:17.405295 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:59:17 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:59:17.710706 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33213
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:33213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:18.069547 20924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:18.069875 20924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:18.070097 20924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:18.078187 20924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:59:18.078311 20924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:18.078361 20924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:18.078419 20924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:59:18.078459 20924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:59:18.088441 20924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:33213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:18.089865 20924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:18.091439 20924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:18.109325 20933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:18.110172 20934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:18.110234 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:18.111281 20924 server_base.cc:1034] running on GCE node
I20260807 02:59:18.112133 20924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:18.113897 20924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:18.115142 20924 hybrid_clock.cc:648] HybridClock initialized: now 1786071558115086 us; error 63 us; skew 500 ppm
I20260807 02:59:18.115528 20924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:18 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071558, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:18.118748 20924 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:59:18.120225 20924 webserver.cc:460] Webserver started at http://127.1.121.126:43473/ using document root <none> and password file <none>
I20260807 02:59:18.121065 20924 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:18.121228 20924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:18.121546 20924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:18.124739 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "9af7655807744da8bc505e64dfd96fcc"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "7a0faf0157ca2cc200d6e16a605ea849"
server_key_iv: "68863f459141ba1a815c3cf029897204"
server_key_version: "encryptionkey@0"
I20260807 02:59:18.125620 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "9af7655807744da8bc505e64dfd96fcc"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "7a0faf0157ca2cc200d6e16a605ea849"
server_key_iv: "68863f459141ba1a815c3cf029897204"
server_key_version: "encryptionkey@0"
I20260807 02:59:18.130892 20924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:59:18.134289 20949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:18.135617 20924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:59:18.135823 20924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "9af7655807744da8bc505e64dfd96fcc"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "7a0faf0157ca2cc200d6e16a605ea849"
server_key_iv: "68863f459141ba1a815c3cf029897204"
server_key_version: "encryptionkey@0"
I20260807 02:59:18.136034 20924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:18.174856 20924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:18.179168 20924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:18.179504 20924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:18.199220 20924 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:33213
I20260807 02:59:18.199240 21034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:33213 every 8 connection(s)
I20260807 02:59:18.200709 20924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:59:18.202507 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 20924
I20260807 02:59:18.202764 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:59:18.203490 1509 external_mini_cluster.cc:1428] Setting key 5025852b7de006e82afccb404a748263
I20260807 02:59:18.205289 21039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 02:59:18 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:18.219417 21039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Bootstrap starting.
I20260807 02:59:18.223644 21039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:18.225431 21039 log.cc:824] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:18.229543 21039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: No bootstrap required, opened a new log
I20260807 02:59:18.230159 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:18.205719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.1.121.126:33213)
0807 02:59:18.206578 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:18.206593 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:18.206644 (+ 51us) server_negotiation.cc:407] Connection header received
0807 02:59:18.208677 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:18.208708 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:18.209287 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:18.209754 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:18.211123 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:18.213205 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:18.214515 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:18.215578 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:18.215776 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:18.220144 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:18.220155 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:18.220174 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:18.220219 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:18.224685 (+ 4466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:18.225502 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:18.225512 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:18.225521 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:18.225761 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:18.226254 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:18.226259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:18.226263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:18.227140 (+ 877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:18.227505 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:18.228129 (+ 624us) server_negotiation.cc:299] Negotiation successful
0807 02:59:18.228635 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":611,"thread_start_us":347,"threads_started":1}
I20260807 02:59:18.238914 21039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:18.239341 21039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:18.239459 21039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af7655807744da8bc505e64dfd96fcc, State: Initialized, Role: FOLLOWER
I20260807 02:59:18.240067 21039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:18.240360 21039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:59:18.240532 21039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:59:18.240751 21039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:18.243247 21039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:18.243821 21039 leader_election.cc:302] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9af7655807744da8bc505e64dfd96fcc; no voters:
I20260807 02:59:18.244477 21039 leader_election.cc:288] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:59:18.244596 21046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:59:18.245450 21046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 1 LEADER]: Becoming Leader. State: Replica: 9af7655807744da8bc505e64dfd96fcc, State: Running, Role: LEADER
I20260807 02:59:18.246029 21046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:18.246874 21039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:59:18.249493 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9af7655807744da8bc505e64dfd96fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } } }
I20260807 02:59:18.249454 21048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af7655807744da8bc505e64dfd96fcc. Latest consensus state: current_term: 1 leader_uuid: "9af7655807744da8bc505e64dfd96fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } } }
I20260807 02:59:18.250052 21048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:18.250080 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:18.256510 21053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:59:18.262899 21053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:59:18.273243 21053 catalog_manager.cc:1347] Generated new cluster ID: 4a4f2cdfd0914f2fabfc114c2b5c308b
I20260807 02:59:18.273394 21053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:59:18.292325 21053 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:59:18.294541 21053 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:59:18.304919 21053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Generated new TSK 0
I20260807 02:59:18.306140 21053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:59:18.377784 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33213
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:18.777225 21079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:18.777518 21079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:18.777616 21079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:18.785192 21079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:18.785308 21079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:18.785429 21079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:59:18.795805 21079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:18.797202 21079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:18.798549 21079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:18.817804 21088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:18.817900 21091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:18.818126 21085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:18.819092 21079 server_base.cc:1034] running on GCE node
I20260807 02:59:18.820242 21079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:18.821961 21079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:18.823390 21079 hybrid_clock.cc:648] HybridClock initialized: now 1786071558823305 us; error 68 us; skew 500 ppm
I20260807 02:59:18.823865 21079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:18 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071558, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:18.828996 21079 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:59:18.831156 21079 webserver.cc:460] Webserver started at http://127.1.121.65:38401/ using document root <none> and password file <none>
I20260807 02:59:18.832238 21079 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:18.832438 21079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:18.832904 21079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:18.837157 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "92c884fcae264409bb21cc38b26cf099"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "1552f0a90a0fc7df064f8e6096fad3b3"
server_key_iv: "edd599a26292141c8bfd44d7d970fa2f"
server_key_version: "encryptionkey@0"
I20260807 02:59:18.838263 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "92c884fcae264409bb21cc38b26cf099"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "1552f0a90a0fc7df064f8e6096fad3b3"
server_key_iv: "edd599a26292141c8bfd44d7d970fa2f"
server_key_version: "encryptionkey@0"
I20260807 02:59:18.846145 21079 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260807 02:59:18.853102 21100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:18.855389 21079 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260807 02:59:18.855732 21079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "92c884fcae264409bb21cc38b26cf099"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "1552f0a90a0fc7df064f8e6096fad3b3"
server_key_iv: "edd599a26292141c8bfd44d7d970fa2f"
server_key_version: "encryptionkey@0"
I20260807 02:59:18.856110 21079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:18.870812 21079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:18.875263 21079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:18.875514 21079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:18.876437 21079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:18.877873 21079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:18.877949 21079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:18.878036 21079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:18.878090 21079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:18.913764 21079 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:43631
I20260807 02:59:18.913874 21259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:43631 every 8 connection(s)
I20260807 02:59:18.915940 21079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:59:18.917232 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 21079
I20260807 02:59:18.917500 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:59:18.918537 1509 external_mini_cluster.cc:1428] Setting key 3f78da832025edf52c65a44abcd0f999
Aug 07 02:59:18 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071558, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:18.948916 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:18.922873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:34739 (local address 127.1.121.126:33213)
0807 02:59:18.923487 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:18.923496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:18.926103 (+ 2607us) server_negotiation.cc:407] Connection header received
0807 02:59:18.930025 (+ 3922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:18.930034 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:18.930145 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:18.930234 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:18.932148 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:18.933327 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:18.935398 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:18.935797 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:18.935900 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:18.940464 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:18.940474 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:18.940482 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:18.940531 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:18.944675 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:18.945724 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:18.945733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:18.945739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:18.945910 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:18.946599 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:18.946607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:18.946614 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:18.947026 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:18.947185 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:18.948047 (+ 862us) server_negotiation.cc:299] Negotiation successful
0807 02:59:18.948315 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
I20260807 02:59:18.949873 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:18.923598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:34739)
0807 02:59:18.925911 (+ 2313us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:18.925977 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:18.929663 (+ 3686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:18.930815 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:18.930823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:18.931337 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:18.931927 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:18.931944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:18.933563 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:18.933568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:18.935199 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:18.935219 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:18.936054 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:18.936065 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:18.936362 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:18.937167 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:18.937196 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:18.940094 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:18.944953 (+ 4859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:18.944963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:18.944980 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:18.945569 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:18.946230 (+ 661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:18.946234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:18.946237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:18.946440 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:18.947166 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:18.947173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:18.947611 (+ 438us) client_negotiation.cc:769] Sending connection context
0807 02:59:18.947869 (+ 258us) client_negotiation.cc:240] Negotiation successful
0807 02:59:18.948378 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1933,"thread_start_us":1688,"threads_started":1}
I20260807 02:59:18.951781 21261 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33213
I20260807 02:59:18.952155 21261 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:18.953177 21261 heartbeater.cc:511] Master 127.1.121.126:33213 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
I20260807 02:59:18.955801 20977 ts_manager.cc:194] Registered new tserver with Master: 92c884fcae264409bb21cc38b26cf099 (127.1.121.65:43631)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
I20260807 02:59:18.958303 20977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:34739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:59:18.992347 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33213
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:19.368597 21275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:19.369100 21275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:19.369305 21275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:19.378129 21275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:19.378239 21275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:19.378427 21275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:59:19.388507 21275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:19.389954 21275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:19.391350 21275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:19.405479 21283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:19.405525 21287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:19.405568 21285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:19.406450 21275 server_base.cc:1034] running on GCE node
I20260807 02:59:19.407130 21275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:19.408195 21275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:19.409674 21275 hybrid_clock.cc:648] HybridClock initialized: now 1786071559409603 us; error 54 us; skew 500 ppm
I20260807 02:59:19.410040 21275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:19 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071559, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:19.413771 21275 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:59:19.415335 21275 webserver.cc:460] Webserver started at http://127.1.121.66:36623/ using document root <none> and password file <none>
I20260807 02:59:19.416078 21275 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:19.416179 21275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:19.416428 21275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:19.419545 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "fc41626422d40bef2ce894a7b8be1b78"
server_key_iv: "c724addfbbe8cd5ba20db2ec51c29a9a"
server_key_version: "encryptionkey@0"
I20260807 02:59:19.420272 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "fc41626422d40bef2ce894a7b8be1b78"
server_key_iv: "c724addfbbe8cd5ba20db2ec51c29a9a"
server_key_version: "encryptionkey@0"
I20260807 02:59:19.425700 21275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 02:59:19.429190 21300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:19.430560 21275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:59:19.430730 21275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "fc41626422d40bef2ce894a7b8be1b78"
server_key_iv: "c724addfbbe8cd5ba20db2ec51c29a9a"
server_key_version: "encryptionkey@0"
I20260807 02:59:19.430922 21275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:19.461086 21275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:19.465473 21275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:19.465749 21275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:19.466691 21275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:19.468039 21275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:19.468116 21275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:19.468201 21275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:19.468245 21275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:19.503295 21275 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:34285
I20260807 02:59:19.503341 21472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:34285 every 8 connection(s)
I20260807 02:59:19.504639 21275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:59:19.514385 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 21275
I20260807 02:59:19.514606 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:59:19.515125 1509 external_mini_cluster.cc:1428] Setting key d66b484e08fe21c506c2be8d92943152
Aug 07 02:59:19 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071559, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:19.527352 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:19.507442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:51355 (local address 127.1.121.126:33213)
0807 02:59:19.507924 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:19.507929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:19.508938 (+ 1009us) server_negotiation.cc:407] Connection header received
0807 02:59:19.511126 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:19.511133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:19.511240 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:19.511376 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:19.513115 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:19.513976 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:19.515841 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:19.516437 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:19.516550 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:19.520997 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:19.521002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:19.521007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:19.521045 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:19.524054 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:19.524868 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:19.524876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:19.524880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:19.525030 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:19.525489 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:19.525497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:19.525503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:19.525849 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:19.526017 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:19.526737 (+ 720us) server_negotiation.cc:299] Negotiation successful
0807 02:59:19.526981 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":270,"threads_started":1}
I20260807 02:59:19.528359 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:19.507914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:51355)
0807 02:59:19.508714 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:19.508756 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:19.510781 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:19.511639 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:19.511648 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:19.512238 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:19.512913 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:19.512936 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:19.514181 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:19.514185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:19.515674 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:19.515688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:19.516718 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:19.516727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:19.517038 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:19.517879 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:19.517911 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:19.520721 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:19.524207 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:19.524216 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:19.524234 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:19.524682 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:19.525149 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:19.525154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:19.525158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:19.525357 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:19.525982 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:19.525991 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:19.526340 (+ 349us) client_negotiation.cc:769] Sending connection context
0807 02:59:19.526626 (+ 286us) client_negotiation.cc:240] Negotiation successful
0807 02:59:19.527179 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":323,"threads_started":1}
I20260807 02:59:19.530004 21473 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33213
I20260807 02:59:19.530400 21473 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:19.531337 21473 heartbeater.cc:511] Master 127.1.121.126:33213 requested a full tablet report, sending...
I20260807 02:59:19.533106 20977 ts_manager.cc:194] Registered new tserver with Master: 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
I20260807 02:59:19.534413 20977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:51355
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:59:19.585976 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33213
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:19.946815 21481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:19.947243 21481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:19.947436 21481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:19.955997 21481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:19.956169 21481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:19.956380 21481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:59:19.961862 21261 heartbeater.cc:503] Master 127.1.121.126:33213 was elected leader, sending a full tablet report...
I20260807 02:59:19.968612 21481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:19.970206 21481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:19.971729 21481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:19.987221 21489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:19.987269 21488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:19.988023 21481 server_base.cc:1034] running on GCE node
W20260807 02:59:19.988099 21493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:19.988814 21481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:19.989938 21481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:19.991175 21481 hybrid_clock.cc:648] HybridClock initialized: now 1786071559991109 us; error 65 us; skew 500 ppm
I20260807 02:59:19.991518 21481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:19 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071559, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:19.994995 21481 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:59:19.996590 21481 webserver.cc:460] Webserver started at http://127.1.121.67:37291/ using document root <none> and password file <none>
I20260807 02:59:19.997471 21481 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:19.997630 21481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:19.997957 21481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:20.001368 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "ef6e9893175c458cbe761e236f021cea"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "0c21ef68e6ac375be49653319d8a6d23"
server_key_iv: "28fc13ea429f1eeb0664d120a5daa5f6"
server_key_version: "encryptionkey@0"
I20260807 02:59:20.002192 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "ef6e9893175c458cbe761e236f021cea"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "0c21ef68e6ac375be49653319d8a6d23"
server_key_iv: "28fc13ea429f1eeb0664d120a5daa5f6"
server_key_version: "encryptionkey@0"
I20260807 02:59:20.007844 21481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 02:59:20.011603 21503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:20.013094 21481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:59:20.013285 21481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "ef6e9893175c458cbe761e236f021cea"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "0c21ef68e6ac375be49653319d8a6d23"
server_key_iv: "28fc13ea429f1eeb0664d120a5daa5f6"
server_key_version: "encryptionkey@0"
I20260807 02:59:20.013505 21481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:20.025755 21481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:20.030604 21481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:20.030933 21481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:20.031970 21481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:20.033475 21481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:20.033645 21481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:20.033818 21481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:20.033895 21481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:20.070865 21481 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:42111
I20260807 02:59:20.070900 21666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:42111 every 8 connection(s)
I20260807 02:59:20.072405 21481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:59:20.078888 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 21481
I20260807 02:59:20.079142 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:59:20.079670 1509 external_mini_cluster.cc:1428] Setting key 260bc542cc861d71cebc791bb7a04709
Aug 07 02:59:20 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071559, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:20.093834 21671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.075169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:60869 (local address 127.1.121.126:33213)
0807 02:59:20.075691 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.075698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.076889 (+ 1191us) server_negotiation.cc:407] Connection header received
0807 02:59:20.078740 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.078746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.078836 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.078962 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:20.080694 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.081766 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.083707 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.084098 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.084208 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.087871 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:20.087876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:20.087879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:20.087913 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:20.090543 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.091328 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.091334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.091337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.091474 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.091925 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.091933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.091936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.092227 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:20.092399 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.093185 (+ 786us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.093413 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":243,"threads_started":1}
I20260807 02:59:20.094983 21672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.075944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:60869)
0807 02:59:20.076678 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:20.076723 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:20.078438 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:20.079207 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:20.079216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:20.079801 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:20.080493 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.080523 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.081949 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.081953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.083569 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.083582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.084303 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.084307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.084504 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.085330 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:20.085354 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:20.087634 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:20.090697 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:20.090707 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:20.090724 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:20.091194 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:20.091593 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:20.091598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:20.091602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:20.091805 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:20.092364 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:20.092373 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:20.092743 (+ 370us) client_negotiation.cc:769] Sending connection context
0807 02:59:20.093085 (+ 342us) client_negotiation.cc:240] Negotiation successful
0807 02:59:20.093638 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":296,"threads_started":1}
I20260807 02:59:20.096542 21668 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33213
I20260807 02:59:20.096941 21668 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:20.097658 21668 heartbeater.cc:511] Master 127.1.121.126:33213 requested a full tablet report, sending...
I20260807 02:59:20.099089 20977 ts_manager.cc:194] Registered new tserver with Master: ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111)
I20260807 02:59:20.099915 20977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:60869
I20260807 02:59:20.108326 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:59:20.126438 21671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.112947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.1.121.126:33213)
0807 02:59:20.113131 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.113136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.113642 (+ 506us) server_negotiation.cc:407] Connection header received
0807 02:59:20.113828 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.113835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.113936 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.114048 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:20.115551 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.116275 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.117821 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.118303 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.118446 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.120981 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:20.120986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:20.120990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:20.121022 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:20.123645 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.124360 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.124364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.124367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.124491 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.124926 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.124937 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.124940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.125265 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:20.125548 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.125872 (+ 324us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.126107 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:59:20.130714 20977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:59:20.137909 20977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:59:20.161639 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.151402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.1.121.67:42111)
0807 02:59:20.152218 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.152228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.152264 (+ 36us) server_negotiation.cc:407] Connection header received
0807 02:59:20.152580 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.152588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.152903 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.153106 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:20.154201 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.155876 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.159046 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.160266 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.160433 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.160703 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.161016 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.161181 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":449,"threads_started":1}
I20260807 02:59:20.161727 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.151234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:42111 (local address 127.0.0.1:36846)
0807 02:59:20.152053 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:20.152098 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:20.152285 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:20.153423 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:20.153433 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:20.153464 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:20.153965 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.153981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.156054 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.156065 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.158810 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.158829 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.160476 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.160482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.160690 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.160749 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 02:59:20.160952 (+ 203us) client_negotiation.cc:240] Negotiation successful
0807 02:59:20.161181 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":4,"thread_start_us":337,"threads_started":1}
I20260807 02:59:20.164777 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.152418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:34285 (local address 127.0.0.1:33966)
0807 02:59:20.153185 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:20.153206 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:20.153325 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:20.155589 (+ 2264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:20.155593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:20.155617 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:20.156021 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.156036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.158297 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.158342 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.161476 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.161545 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.163830 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.163836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.164029 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.164079 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 02:59:20.164155 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 02:59:20.164301 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":60,"thread_start_us":743,"threads_started":1}
I20260807 02:59:20.165128 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.152979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.1.121.66:34285)
0807 02:59:20.154979 (+ 2000us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.154989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.155018 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:59:20.155107 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.155116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.155377 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.156240 (+ 863us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:20.156492 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.158172 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.161738 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.163577 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.164341 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.164517 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.164588 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.164764 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":1320,"threads_started":1}
I20260807 02:59:20.165167 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.153144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50172)
0807 02:59:20.153965 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:20.153982 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:20.154103 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:20.156002 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:20.156007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:20.156038 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:20.156464 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.156473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.158297 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.158303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.161506 (+ 3203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.161518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.164084 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.164090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.164272 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.164301 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 02:59:20.164382 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 02:59:20.164525 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":44,"thread_start_us":1011,"threads_started":1}
I20260807 02:59:20.166059 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.153932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.1.121.65:43631)
0807 02:59:20.155143 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.155152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.155181 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:59:20.155286 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.155294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.155547 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.155766 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:20.156702 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.158085 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.161728 (+ 3643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.163502 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.163665 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.163933 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.165552 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.165739 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":874,"threads_started":1}
I20260807 02:59:20.166239 21573 tablet_service.cc:1467] Processing CreateTablet for tablet 0d095415c1e34439a4acf888be476070 (DEFAULT_TABLE table=test-table [id=ae7d09160d4643dcb04bcb810ccce682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:20.167887 21573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d095415c1e34439a4acf888be476070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:20.169236 21166 tablet_service.cc:1467] Processing CreateTablet for tablet 0d095415c1e34439a4acf888be476070 (DEFAULT_TABLE table=test-table [id=ae7d09160d4643dcb04bcb810ccce682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:20.169312 21367 tablet_service.cc:1467] Processing CreateTablet for tablet 0d095415c1e34439a4acf888be476070 (DEFAULT_TABLE table=test-table [id=ae7d09160d4643dcb04bcb810ccce682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:20.170629 21166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d095415c1e34439a4acf888be476070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:20.170718 21367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d095415c1e34439a4acf888be476070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:20.184118 21704 tablet_bootstrap.cc:492] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: Bootstrap starting.
I20260807 02:59:20.185966 21705 tablet_bootstrap.cc:492] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: Bootstrap starting.
I20260807 02:59:20.189095 21705 tablet_bootstrap.cc:654] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:20.189695 21706 tablet_bootstrap.cc:492] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Bootstrap starting.
I20260807 02:59:20.190734 21705 log.cc:824] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:20.192502 21704 tablet_bootstrap.cc:654] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:20.193001 21706 tablet_bootstrap.cc:654] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:20.193833 21704 log.cc:824] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:20.194471 21706 log.cc:824] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:20.196280 21705 tablet_bootstrap.cc:492] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: No bootstrap required, opened a new log
I20260807 02:59:20.196591 21705 ts_tablet_manager.cc:1397] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260807 02:59:20.197256 21704 tablet_bootstrap.cc:492] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: No bootstrap required, opened a new log
I20260807 02:59:20.197577 21704 ts_tablet_manager.cc:1397] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20260807 02:59:20.197824 21706 tablet_bootstrap.cc:492] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: No bootstrap required, opened a new log
I20260807 02:59:20.198122 21706 ts_tablet_manager.cc:1397] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 02:59:20.209043 21705 raft_consensus.cc:348] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } }
I20260807 02:59:20.209600 21705 raft_consensus.cc:374] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:20.209712 21705 raft_consensus.cc:729] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6e9893175c458cbe761e236f021cea, State: Initialized, Role: FOLLOWER
I20260807 02:59:20.209854 21704 raft_consensus.cc:348] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } }
I20260807 02:59:20.209854 21706 raft_consensus.cc:348] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } }
I20260807 02:59:20.210318 21706 raft_consensus.cc:374] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:20.210319 21704 raft_consensus.cc:374] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:20.210433 21704 raft_consensus.cc:729] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c884fcae264409bb21cc38b26cf099, State: Initialized, Role: FOLLOWER
I20260807 02:59:20.210433 21706 raft_consensus.cc:729] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f7d38fd7a4440ab6ae9d2ebac05ad9, State: Initialized, Role: FOLLOWER
I20260807 02:59:20.210603 21705 consensus_queue.cc:260] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } }
I20260807 02:59:20.211134 21704 consensus_queue.cc:260] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } }
I20260807 02:59:20.211165 21706 consensus_queue.cc:260] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } }
I20260807 02:59:20.222563 21705 ts_tablet_manager.cc:1428] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: Time spent starting tablet: real 0.026s user 0.017s sys 0.007s
I20260807 02:59:20.222909 21704 ts_tablet_manager.cc:1428] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: Time spent starting tablet: real 0.025s user 0.022s sys 0.002s
I20260807 02:59:20.222939 21668 heartbeater.cc:503] Master 127.1.121.126:33213 was elected leader, sending a full tablet report...
I20260807 02:59:20.223209 21706 ts_tablet_manager.cc:1428] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Time spent starting tablet: real 0.025s user 0.017s sys 0.008s
I20260807 02:59:20.224751 21473 heartbeater.cc:503] Master 127.1.121.126:33213 was elected leader, sending a full tablet report...
W20260807 02:59:20.256968 21475 tablet.cc:2369] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:20.280947 21717 raft_consensus.cc:482] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:20.281322 21717 raft_consensus.cc:504] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } }
I20260807 02:59:20.283008 21717 leader_election.cc:288] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111), 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
I20260807 02:59:20.290083 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.283386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:38245 (local address 127.1.121.67:42111)
0807 02:59:20.283578 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.283583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.284193 (+ 610us) server_negotiation.cc:407] Connection header received
0807 02:59:20.284568 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.284574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.284653 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.284737 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:20.285517 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.286269 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.288170 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.289037 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.289247 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.289325 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.289680 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.289813 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:59:20.290161 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.284303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:58607 (local address 127.1.121.66:34285)
0807 02:59:20.284522 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.284531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.284999 (+ 468us) server_negotiation.cc:407] Connection header received
0807 02:59:20.285237 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.285242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.285301 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.285403 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:20.285906 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.286586 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.288333 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.289187 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.289363 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.289448 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.289769 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.289896 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:59:20.290251 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.283303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:42111 (local address 127.1.121.65:38245)
0807 02:59:20.284085 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:20.284109 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:20.284337 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:20.284991 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:20.284994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:20.285018 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:20.285361 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.285369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.286408 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.286416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.287996 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.288009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.289255 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.289261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.289506 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.289533 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 02:59:20.289649 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 02:59:20.289793 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":95,"thread_start_us":418,"threads_started":1}
I20260807 02:59:20.290710 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.284168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:34285 (local address 127.1.121.65:58607)
0807 02:59:20.284887 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:20.284908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:20.285055 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:20.285419 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:20.285423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:20.285448 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:20.285754 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.285763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.286743 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.286748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.288167 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:20.288180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.289413 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.289419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:20.289608 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.289630 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 02:59:20.289739 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 02:59:20.289865 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":24,"thread_start_us":353,"threads_started":1}
I20260807 02:59:20.291435 21606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d095415c1e34439a4acf888be476070" candidate_uuid: "92c884fcae264409bb21cc38b26cf099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6e9893175c458cbe761e236f021cea" is_pre_election: true
I20260807 02:59:20.291693 21399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d095415c1e34439a4acf888be476070" candidate_uuid: "92c884fcae264409bb21cc38b26cf099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" is_pre_election: true
I20260807 02:59:20.292011 21606 raft_consensus.cc:2454] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c884fcae264409bb21cc38b26cf099 in term 0.
I20260807 02:59:20.292244 21399 raft_consensus.cc:2454] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c884fcae264409bb21cc38b26cf099 in term 0.
I20260807 02:59:20.292766 21105 leader_election.cc:302] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92c884fcae264409bb21cc38b26cf099, ef6e9893175c458cbe761e236f021cea; no voters:
I20260807 02:59:20.293210 21717 raft_consensus.cc:2781] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:59:20.293331 21717 raft_consensus.cc:482] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:59:20.293484 21717 raft_consensus.cc:3037] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:20.295796 21717 raft_consensus.cc:504] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } }
I20260807 02:59:20.296494 21717 leader_election.cc:288] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 1 election: Requested vote from peers ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111), 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
I20260807 02:59:20.297061 21606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d095415c1e34439a4acf888be476070" candidate_uuid: "92c884fcae264409bb21cc38b26cf099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6e9893175c458cbe761e236f021cea"
I20260807 02:59:20.297098 21399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d095415c1e34439a4acf888be476070" candidate_uuid: "92c884fcae264409bb21cc38b26cf099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9"
I20260807 02:59:20.297290 21606 raft_consensus.cc:3037] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:20.297291 21399 raft_consensus.cc:3037] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:20.299467 21399 raft_consensus.cc:2454] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c884fcae264409bb21cc38b26cf099 in term 1.
I20260807 02:59:20.299639 21606 raft_consensus.cc:2454] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c884fcae264409bb21cc38b26cf099 in term 1.
I20260807 02:59:20.299917 21105 leader_election.cc:302] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f7d38fd7a4440ab6ae9d2ebac05ad9, 92c884fcae264409bb21cc38b26cf099; no voters:
I20260807 02:59:20.300206 21717 raft_consensus.cc:2781] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:59:20.300702 21717 raft_consensus.cc:686] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 1 LEADER]: Becoming Leader. State: Replica: 92c884fcae264409bb21cc38b26cf099, State: Running, Role: LEADER
I20260807 02:59:20.301277 21717 consensus_queue.cc:237] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } }
I20260807 02:59:20.306103 20972 catalog_manager.cc:5495] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92c884fcae264409bb21cc38b26cf099 (127.1.121.65). New cstate: current_term: 1 leader_uuid: "92c884fcae264409bb21cc38b26cf099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } health_report { overall_health: HEALTHY } } }
W20260807 02:59:20.324276 21670 tablet.cc:2369] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:20.356134 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.350701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.1.121.65:43631)
0807 02:59:20.350951 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.350957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.350976 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:59:20.351079 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.351083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.351165 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.351329 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:59:20.351650 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.352453 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.353925 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.354388 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.354491 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.355001 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:59:20.355538 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:59:20.355660 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.355747 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.355816 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 02:59:20.365026 21606 raft_consensus.cc:1261] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [term 1 FOLLOWER]: Refusing update from remote peer 92c884fcae264409bb21cc38b26cf099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:20.365038 21399 raft_consensus.cc:1261] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 1 FOLLOWER]: Refusing update from remote peer 92c884fcae264409bb21cc38b26cf099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:20.366034 21717 consensus_queue.cc:1035] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:20.366385 21722 consensus_queue.cc:1035] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:20.376497 21728 mvcc.cc:205] Tried to move back new op lower bound from 7315749111244042240 to 7315749110998233088. Current Snapshot: MvccSnapshot[applied={T|T < 7315749111244042240}]
I20260807 02:59:20.377521 21730 mvcc.cc:205] Tried to move back new op lower bound from 7315749111244042240 to 7315749110998233088. Current Snapshot: MvccSnapshot[applied={T|T < 7315749111244042240}]
I20260807 02:59:20.386122 21729 mvcc.cc:205] Tried to move back new op lower bound from 7315749111244042240 to 7315749110998233088. Current Snapshot: MvccSnapshot[applied={T|T < 7315749111244042240}]
I20260807 02:59:20.401927 20970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:59:20.402227 20970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:59:20.406203 20970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Sending DeleteTablet for 3 replicas of tablet 0d095415c1e34439a4acf888be476070
I20260807 02:59:20.407644 21367 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d095415c1e34439a4acf888be476070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:20 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:33966
I20260807 02:59:20.407723 21573 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d095415c1e34439a4acf888be476070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:20 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:36846
I20260807 02:59:20.408003 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 21079
I20260807 02:59:20.408464 21166 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d095415c1e34439a4acf888be476070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:20 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:50172
I20260807 02:59:20.417646 21747 tablet_replica.cc:330] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: stopping tablet replica
I20260807 02:59:20.418319 21747 raft_consensus.cc:2229] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:59:20.419001 21747 raft_consensus.cc:2258] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:20.421967 21747 ts_tablet_manager.cc:1905] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:20.423708 21749 tablet_replica.cc:330] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: stopping tablet replica
I20260807 02:59:20.424247 21749 raft_consensus.cc:2229] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:59:20.425022 21749 raft_consensus.cc:2258] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:20.427558 21749 ts_tablet_manager.cc:1905] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:20.427748 21746 tablet_replica.cc:330] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: stopping tablet replica
I20260807 02:59:20.428166 21746 raft_consensus.cc:2229] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:59:20.428694 21746 raft_consensus.cc:2258] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 07 02:59:20 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:59:20.434689 21746 ts_tablet_manager.cc:1905] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:20.435235 21749 ts_tablet_manager.cc:1918] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:20.435441 21749 log.cc:1196] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/0d095415c1e34439a4acf888be476070
I20260807 02:59:20.436071 21749 ts_tablet_manager.cc:1939] T 0d095415c1e34439a4acf888be476070 P 92c884fcae264409bb21cc38b26cf099: Deleting consensus metadata
I20260807 02:59:20.438021 20951 catalog_manager.cc:4841] TS 92c884fcae264409bb21cc38b26cf099 (127.1.121.65:43631): tablet 0d095415c1e34439a4acf888be476070 (table test-table [id=ae7d09160d4643dcb04bcb810ccce682]) successfully deleted
I20260807 02:59:20.441140 21746 ts_tablet_manager.cc:1918] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:20.441311 21746 log.cc:1196] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/0d095415c1e34439a4acf888be476070
I20260807 02:59:20.441805 21747 ts_tablet_manager.cc:1918] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:20.441890 21746 ts_tablet_manager.cc:1939] T 0d095415c1e34439a4acf888be476070 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Deleting consensus metadata
I20260807 02:59:20.441998 21747 log.cc:1196] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/0d095415c1e34439a4acf888be476070
I20260807 02:59:20.442575 21747 ts_tablet_manager.cc:1939] T 0d095415c1e34439a4acf888be476070 P ef6e9893175c458cbe761e236f021cea: Deleting consensus metadata
I20260807 02:59:20.443814 20956 catalog_manager.cc:4841] TS 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285): tablet 0d095415c1e34439a4acf888be476070 (table test-table [id=ae7d09160d4643dcb04bcb810ccce682]) successfully deleted
I20260807 02:59:20.444278 20956 catalog_manager.cc:4841] TS ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111): tablet 0d095415c1e34439a4acf888be476070 (table test-table [id=ae7d09160d4643dcb04bcb810ccce682]) successfully deleted
I20260807 02:59:20.445422 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.411607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.1.121.65:43631)
0807 02:59:20.412005 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.412015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.421953 (+ 9938us) server_negotiation.cc:407] Connection header received
0807 02:59:20.422550 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.422559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.422675 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.422771 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:20.425475 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.427042 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.429077 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.429904 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.430022 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.436800 (+ 6778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:20.436817 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:20.436854 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:20.436919 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:20.441246 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.442403 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.442412 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.442416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.442604 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.443179 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.443184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.443187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.443651 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:20.444022 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.444723 (+ 701us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.445017 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260807 02:59:20.446223 21253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:20.551589 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 21079
I20260807 02:59:20.577443 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 21275
Aug 07 02:59:20 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:59:20.598477 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.578218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.1.121.66:34285)
0807 02:59:20.578413 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.578421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.578479 (+ 58us) server_negotiation.cc:407] Connection header received
0807 02:59:20.578740 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.578744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.578829 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.578950 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:20.580296 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.581116 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.582440 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.582837 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.582981 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.592363 (+ 9382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:20.592381 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:20.592402 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:20.592450 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:20.595388 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.596179 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.596188 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.596191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.596328 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.596738 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.596744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.596749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.597218 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:20.597559 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.597871 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.598099 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 02:59:20.599164 21468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:20.691246 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 21275
I20260807 02:59:20.716789 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 21481
Aug 07 02:59:20 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:59:20.731918 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:20.717770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.1.121.67:42111)
0807 02:59:20.717958 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:20.717963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:20.717990 (+ 27us) server_negotiation.cc:407] Connection header received
0807 02:59:20.718092 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:20.718096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:20.718164 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:20.718256 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:20.719578 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.720321 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.721714 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:20.722094 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:20.722203 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:20.726444 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:20.726461 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:20.726480 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:20.726523 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:20.729413 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.730063 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.730070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.730073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.730216 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:20.730528 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:20.730531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:20.730533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:20.730892 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:20.731112 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:20.731356 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 02:59:20.731518 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 02:59:20.732585 21659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:20.823105 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 21481
I20260807 02:59:20.846903 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 20924
I20260807 02:59:20.848163 21028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:20.959923 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 20924
I20260807 02:59:20.990484 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33213
--webserver_interface=127.1.121.126
--webserver_port=43473
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:33213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:21.361120 21759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:21.361483 21759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:21.361680 21759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:21.369227 21759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:59:21.369342 21759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:21.369418 21759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:21.369508 21759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:59:21.369601 21759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:59:21.379740 21759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:33213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=43473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:21.381151 21759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:21.383023 21759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:21.411641 21769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:21.411651 21772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:21.411641 21768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:21.413070 21759 server_base.cc:1034] running on GCE node
I20260807 02:59:21.413982 21759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:21.415591 21759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:21.416818 21759 hybrid_clock.cc:648] HybridClock initialized: now 1786071561416757 us; error 31 us; skew 500 ppm
I20260807 02:59:21.417316 21759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:21 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071561, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:21.426769 21759 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:59:21.428534 21759 webserver.cc:460] Webserver started at http://127.1.121.126:43473/ using document root <none> and password file <none>
I20260807 02:59:21.429526 21759 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:21.429697 21759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:21.435938 21759 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.005s
I20260807 02:59:21.438982 21781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:21.440423 21759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:59:21.440625 21759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "9af7655807744da8bc505e64dfd96fcc"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "7a0faf0157ca2cc200d6e16a605ea849"
server_key_iv: "68863f459141ba1a815c3cf029897204"
server_key_version: "encryptionkey@0"
I20260807 02:59:21.441777 21759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:21.456596 21759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:21.460664 21759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:21.460991 21759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:21.480666 21759 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:33213
I20260807 02:59:21.480690 21859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:33213 every 8 connection(s)
I20260807 02:59:21.482297 21759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:59:21.489629 21862 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:59:21.492152 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 21759
I20260807 02:59:21.493242 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:43631
--local_ip_for_outbound_sockets=127.1.121.65
--tserver_master_addrs=127.1.121.126:33213
--webserver_port=38401
--webserver_interface=127.1.121.65
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:59:21.493742 21862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Bootstrap starting.
I20260807 02:59:21.518950 21862 log.cc:824] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:21.532560 21862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:59:21.533183 21862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Bootstrap complete.
I20260807 02:59:21.542892 21862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:21.543744 21862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af7655807744da8bc505e64dfd96fcc, State: Initialized, Role: FOLLOWER
I20260807 02:59:21.544468 21862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:21.544725 21862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:59:21.544919 21862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:59:21.545149 21862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:59:21.549448 21862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:21.549983 21862 leader_election.cc:302] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9af7655807744da8bc505e64dfd96fcc; no voters:
I20260807 02:59:21.550676 21862 leader_election.cc:288] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 02:59:21.550834 21866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 2 FOLLOWER]: Leader election won for term 2
I20260807 02:59:21.552278 21866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 2 LEADER]: Becoming Leader. State: Replica: 9af7655807744da8bc505e64dfd96fcc, State: Running, Role: LEADER
I20260807 02:59:21.552924 21866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:21.553290 21862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:59:21.555799 21869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af7655807744da8bc505e64dfd96fcc. Latest consensus state: current_term: 2 leader_uuid: "9af7655807744da8bc505e64dfd96fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } } }
I20260807 02:59:21.556159 21869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:21.556740 21868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9af7655807744da8bc505e64dfd96fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } } }
I20260807 02:59:21.557070 21868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:21.562320 21876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:59:21.573161 21876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:59:21.575565 21876 catalog_manager.cc:1259] Loaded cluster ID: 4a4f2cdfd0914f2fabfc114c2b5c308b
I20260807 02:59:21.575697 21876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:59:21.580273 21876 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 02:59:21.597450 21876 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 02:59:21.605722 21876 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:59:21.607115 21876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Loaded TSK: 0
I20260807 02:59:21.608299 21876 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 02:59:21.899272 21864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:21.899637 21864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:21.899832 21864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:21.907361 21864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:21.907526 21864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:21.907722 21864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:59:21.918277 21864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:43631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=38401
--tserver_master_addrs=127.1.121.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:21.919720 21864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:21.921192 21864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:21.936618 21915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:21.936702 21917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:21.937048 21920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:21.937825 21864 server_base.cc:1034] running on GCE node
I20260807 02:59:21.938637 21864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:21.939761 21864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:21.940984 21864 hybrid_clock.cc:648] HybridClock initialized: now 1786071561940923 us; error 55 us; skew 500 ppm
I20260807 02:59:21.941334 21864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:21 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071561, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:21.944751 21864 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:59:21.946753 21864 webserver.cc:460] Webserver started at http://127.1.121.65:38401/ using document root <none> and password file <none>
I20260807 02:59:21.947541 21864 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:21.947713 21864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:21.953707 21864 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 02:59:21.956948 21932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:21.958480 21864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:59:21.958690 21864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "92c884fcae264409bb21cc38b26cf099"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "1552f0a90a0fc7df064f8e6096fad3b3"
server_key_iv: "edd599a26292141c8bfd44d7d970fa2f"
server_key_version: "encryptionkey@0"
I20260807 02:59:21.959748 21864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:21.982625 21864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:21.987314 21864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:21.987640 21864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:21.988744 21864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:21.990245 21864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:21.990384 21864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:21.990532 21864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:21.990625 21864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:22.026084 21864 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:43631
I20260807 02:59:22.026180 22103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:43631 every 8 connection(s)
I20260807 02:59:22.027634 21864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:59:22.037472 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 21864
I20260807 02:59:22.038326 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:34285
--local_ip_for_outbound_sockets=127.1.121.66
--tserver_master_addrs=127.1.121.126:33213
--webserver_port=36623
--webserver_interface=127.1.121.66
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:59:22 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071561, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:22.055100 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:22.031148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:45507)
0807 02:59:22.031935 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:22.031981 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:22.033743 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:22.034992 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:22.034999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:22.035492 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:22.036032 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:22.036049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:22.037796 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:22.037802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:22.040298 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:22.040321 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:22.041477 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:22.041491 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:22.041852 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:22.042659 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:22.042708 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:22.046213 (+ 3505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:22.049926 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:22.049939 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:22.049960 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:22.050684 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:22.051150 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:22.051159 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:22.051166 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:22.051445 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:22.052507 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:22.052513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:22.052924 (+ 411us) client_negotiation.cc:769] Sending connection context
0807 02:59:22.053205 (+ 281us) client_negotiation.cc:240] Negotiation successful
0807 02:59:22.053775 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":318,"threads_started":1}
I20260807 02:59:22.055279 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:22.030802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:45507 (local address 127.1.121.126:33213)
0807 02:59:22.031430 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:22.031450 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:22.032123 (+ 673us) server_negotiation.cc:407] Connection header received
0807 02:59:22.034230 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:22.034242 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:22.034562 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:22.034856 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:22.036235 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:22.037611 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:22.040470 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:22.041206 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:22.041346 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:22.046572 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:22.046582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:22.046598 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:22.046636 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:22.049756 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:22.050826 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:22.050833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:22.050838 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:22.051010 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:22.051599 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:22.051605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:22.051609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:22.052341 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:22.052541 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:22.053471 (+ 930us) server_negotiation.cc:299] Negotiation successful
0807 02:59:22.053990 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":230,"threads_started":1}
I20260807 02:59:22.058650 22104 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33213
I20260807 02:59:22.059001 22104 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:22.059828 22104 heartbeater.cc:511] Master 127.1.121.126:33213 requested a full tablet report, sending...
I20260807 02:59:22.062546 21805 ts_manager.cc:194] Registered new tserver with Master: 92c884fcae264409bb21cc38b26cf099 (127.1.121.65:43631)
I20260807 02:59:22.064303 21805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:45507
W20260807 02:59:22.401840 22112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:22.402177 22112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:22.402349 22112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:22.409730 22112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:22.409893 22112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:22.410089 22112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:59:22.419986 22112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:34285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=36623
--tserver_master_addrs=127.1.121.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:22.421403 22112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:22.422798 22112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:22.436267 22119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:22.436231 22120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:22.436306 22123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:22.437163 22112 server_base.cc:1034] running on GCE node
I20260807 02:59:22.437803 22112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:22.438776 22112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:22.439975 22112 hybrid_clock.cc:648] HybridClock initialized: now 1786071562439930 us; error 37 us; skew 500 ppm
I20260807 02:59:22.440299 22112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:22 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071562, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:22.443733 22112 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:59:22.445508 22112 webserver.cc:460] Webserver started at http://127.1.121.66:36623/ using document root <none> and password file <none>
I20260807 02:59:22.446262 22112 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:22.446376 22112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:22.452047 22112 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:59:22.455134 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:22.456534 22112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 02:59:22.456733 22112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "fc41626422d40bef2ce894a7b8be1b78"
server_key_iv: "c724addfbbe8cd5ba20db2ec51c29a9a"
server_key_version: "encryptionkey@0"
I20260807 02:59:22.457792 22112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:22.472143 22112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:22.476634 22112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:22.477043 22112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:22.478233 22112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:22.479693 22112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:22.479815 22112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:22.479980 22112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:22.480073 22112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:22.518325 22112 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:34285
I20260807 02:59:22.518549 22315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:34285 every 8 connection(s)
I20260807 02:59:22.519939 22112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:59:22.524718 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 22112
I20260807 02:59:22.525573 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:42111
--local_ip_for_outbound_sockets=127.1.121.67
--tserver_master_addrs=127.1.121.126:33213
--webserver_port=37291
--webserver_interface=127.1.121.67
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:59:22 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071562, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:22.544414 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:22.522783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:33847 (local address 127.1.121.126:33213)
0807 02:59:22.522988 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:22.522993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:22.524387 (+ 1394us) server_negotiation.cc:407] Connection header received
0807 02:59:22.527312 (+ 2925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:22.527317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:22.527406 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:22.527512 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:22.529707 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:22.530663 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:22.533313 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:22.533770 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:22.533876 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:22.538520 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:22.538525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:22.538528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:22.538563 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:22.541320 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:22.542047 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:22.542052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:22.542060 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:22.542184 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:22.542622 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:22.542627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:22.542630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:22.542902 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:22.543070 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:22.543805 (+ 735us) server_negotiation.cc:299] Negotiation successful
0807 02:59:22.544072 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 02:59:22.545528 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:22.523364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:33847)
0807 02:59:22.524133 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:22.524212 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:22.526948 (+ 2736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:22.527799 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:22.527814 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:22.528609 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:22.529494 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:22.529525 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:22.530831 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:22.530839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:22.533143 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:22.533161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:22.534007 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:22.534026 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:22.534361 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:22.535233 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:22.535268 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:22.538261 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:22.541462 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:22.541472 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:22.541490 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:22.541914 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:22.542299 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:22.542304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:22.542307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:22.542496 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:22.543044 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:22.543052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:22.543393 (+ 341us) client_negotiation.cc:769] Sending connection context
0807 02:59:22.543660 (+ 267us) client_negotiation.cc:240] Negotiation successful
0807 02:59:22.544258 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":283,"threads_started":1}
I20260807 02:59:22.547320 22320 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33213
I20260807 02:59:22.547734 22320 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:22.548719 22320 heartbeater.cc:511] Master 127.1.121.126:33213 requested a full tablet report, sending...
I20260807 02:59:22.550585 21805 ts_manager.cc:194] Registered new tserver with Master: 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
I20260807 02:59:22.552000 21805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:33847
W20260807 02:59:22.886907 22324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:22.887208 22324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:22.887311 22324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:22.894678 22324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:22.894802 22324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:22.894918 22324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:59:22.905889 22324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:42111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=37291
--tserver_master_addrs=127.1.121.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:22.907234 22324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:22.908666 22324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:22.923161 22331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:22.923240 22332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:22.923312 22334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:22.924484 22324 server_base.cc:1034] running on GCE node
I20260807 02:59:22.925298 22324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:22.926348 22324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:22.927584 22324 hybrid_clock.cc:648] HybridClock initialized: now 1786071562927539 us; error 28 us; skew 500 ppm
I20260807 02:59:22.927960 22324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:22 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071562, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:22.931329 22324 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:59:22.932986 22324 webserver.cc:460] Webserver started at http://127.1.121.67:37291/ using document root <none> and password file <none>
I20260807 02:59:22.933712 22324 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:22.933861 22324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:22.939260 22324 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:59:22.942296 22348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:22.943795 22324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:59:22.943974 22324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "ef6e9893175c458cbe761e236f021cea"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "0c21ef68e6ac375be49653319d8a6d23"
server_key_iv: "28fc13ea429f1eeb0664d120a5daa5f6"
server_key_version: "encryptionkey@0"
I20260807 02:59:22.945133 22324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:22.969364 22324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:22.973446 22324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:22.973762 22324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:22.974786 22324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:22.976189 22324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:22.976316 22324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:22.976455 22324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:22.976545 22324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:23.014473 22324 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:42111
I20260807 02:59:23.014568 22526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:42111 every 8 connection(s)
I20260807 02:59:23.016018 22324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:59:23.020354 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 22324
Aug 07 02:59:23 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071562, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:23.039248 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.018937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:50387 (local address 127.1.121.126:33213)
0807 02:59:23.019196 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.019203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.020416 (+ 1213us) server_negotiation.cc:407] Connection header received
0807 02:59:23.023144 (+ 2728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.023151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.023258 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.023383 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:23.025118 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.025944 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.027585 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.028020 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.028168 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.032501 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:23.032506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:23.032510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:23.032543 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:23.035747 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.036575 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.036581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.036585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.036746 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.037270 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.037276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.037281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.037604 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:23.037779 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.038598 (+ 819us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.038825 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260807 02:59:23.040318 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.019388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:50387)
0807 02:59:23.020208 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.020258 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:23.022839 (+ 2581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:23.023655 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:23.023664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:23.024223 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:23.024930 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.024957 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.026117 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.026123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.027401 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.027417 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.028283 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.028293 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.028553 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.029334 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:23.029360 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:23.032243 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:23.035904 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:23.035914 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:23.035940 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:23.036424 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:23.036894 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:23.036900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:23.036904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:23.037129 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:23.037752 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:23.037761 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:23.038142 (+ 381us) client_negotiation.cc:769] Sending connection context
0807 02:59:23.038479 (+ 337us) client_negotiation.cc:240] Negotiation successful
0807 02:59:23.039055 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":292,"threads_started":1}
I20260807 02:59:23.040606 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.021438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.1.121.126:33213)
0807 02:59:23.021955 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.021963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.022387 (+ 424us) server_negotiation.cc:407] Connection header received
0807 02:59:23.022609 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.022613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.022684 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.022812 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:23.024397 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.025295 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.030787 (+ 5492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.031509 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.031650 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.034846 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:23.034851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:23.034854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:23.034894 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:23.037661 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.038342 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.038354 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.038357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.038467 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.038864 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.038869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.038872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.039146 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:23.039431 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.039952 (+ 521us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.040183 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":388,"threads_started":1}
I20260807 02:59:23.042088 22529 heartbeater.cc:347] Connected to a master server at 127.1.121.126:33213
I20260807 02:59:23.042487 22529 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:23.043413 22529 heartbeater.cc:511] Master 127.1.121.126:33213 requested a full tablet report, sending...
I20260807 02:59:23.045105 21804 ts_manager.cc:194] Registered new tserver with Master: ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111)
I20260807 02:59:23.046383 21804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:50387
I20260807 02:59:23.055821 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:59:23.063284 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.056958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.1.121.126:33213)
0807 02:59:23.057236 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.057243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.057537 (+ 294us) server_negotiation.cc:407] Connection header received
0807 02:59:23.057817 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.057821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.057956 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.058120 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:59:23.058575 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.059291 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.060973 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.061425 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.061602 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.062111 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:59:23.062664 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:59:23.062947 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.062993 (+ 46us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.063071 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 02:59:23.067519 22104 heartbeater.cc:503] Master 127.1.121.126:33213 was elected leader, sending a full tablet report...
I20260807 02:59:23.064388 21804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:59:23.072219 21804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:59:23.094907 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.084706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.1.121.67:42111)
0807 02:59:23.086691 (+ 1985us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.086702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.086740 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:59:23.086846 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.086854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.087110 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.087965 (+ 855us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:23.088300 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.090005 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.092457 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.093647 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.093820 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.094069 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.094373 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.094546 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1799,"thread_start_us":1617,"threads_started":1}
I20260807 02:59:23.094949 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.084392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:42111 (local address 127.0.0.1:36864)
0807 02:59:23.085368 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.085420 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:23.085591 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:23.087345 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:23.087353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:23.087383 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:23.087926 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.087938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.090195 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.090200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.092227 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.092243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.093892 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.093898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.094128 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.094186 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 02:59:23.094336 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 02:59:23.094542 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":9,"thread_start_us":2023,"threads_started":1}
I20260807 02:59:23.096582 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.084566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50198)
0807 02:59:23.086686 (+ 2120us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.086704 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:23.086894 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:23.087829 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:23.087836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:23.087869 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:23.089168 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.089181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.091056 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.091062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.093657 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.093672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.095765 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.095771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.095937 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.095969 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 02:59:23.096062 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 02:59:23.096203 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2001,"mutex_wait_us":858,"thread_start_us":857,"threads_started":1}
I20260807 02:59:23.096683 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.085252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.1.121.65:43631)
0807 02:59:23.086244 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.086256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.086797 (+ 541us) server_negotiation.cc:407] Connection header received
0807 02:59:23.087069 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.087077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.087309 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.087469 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:23.089362 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.090900 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.093858 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.095411 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.095533 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.095781 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.096160 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.096336 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":741,"threads_started":1}
I20260807 02:59:23.098567 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.087103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.1.121.66:34285)
0807 02:59:23.089558 (+ 2455us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.089567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.089629 (+ 62us) server_negotiation.cc:407] Connection header received
0807 02:59:23.089699 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.089704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.089852 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.090048 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:23.090728 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.092270 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.094784 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.096612 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.096964 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.097170 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.098043 (+ 873us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.098226 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2313,"thread_start_us":2252,"threads_started":1}
I20260807 02:59:23.099087 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.086829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:34285 (local address 127.0.0.1:33986)
0807 02:59:23.088500 (+ 1671us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.088519 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:23.088625 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:23.090035 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:23.090039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:23.090064 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:23.090531 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.090542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.092441 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.092447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.094583 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.094599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.097322 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.097329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.097555 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.097590 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 02:59:23.097685 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 02:59:23.097823 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"mutex_wait_us":2,"thread_start_us":1396,"threads_started":1}
I20260807 02:59:23.099412 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 0573bd29c10c4c1db10946993ab0f6a5 (DEFAULT_TABLE table=test-table [id=1f61191c4c304e589732ac9f4cd22f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:23.100386 22003 tablet_service.cc:1467] Processing CreateTablet for tablet 0573bd29c10c4c1db10946993ab0f6a5 (DEFAULT_TABLE table=test-table [id=1f61191c4c304e589732ac9f4cd22f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:23.100941 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0573bd29c10c4c1db10946993ab0f6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:23.101613 22003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0573bd29c10c4c1db10946993ab0f6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:23.102556 22209 tablet_service.cc:1467] Processing CreateTablet for tablet 0573bd29c10c4c1db10946993ab0f6a5 (DEFAULT_TABLE table=test-table [id=1f61191c4c304e589732ac9f4cd22f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:23.103813 22209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0573bd29c10c4c1db10946993ab0f6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:23.113950 22547 tablet_bootstrap.cc:492] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Bootstrap starting.
I20260807 02:59:23.118062 22548 tablet_bootstrap.cc:492] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: Bootstrap starting.
I20260807 02:59:23.119278 22547 tablet_bootstrap.cc:654] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:23.120196 22550 tablet_bootstrap.cc:492] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Bootstrap starting.
I20260807 02:59:23.121157 22547 log.cc:824] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:23.121873 22548 tablet_bootstrap.cc:654] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:23.122956 22548 log.cc:824] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:23.123622 22550 tablet_bootstrap.cc:654] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:23.124572 22547 tablet_bootstrap.cc:492] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: No bootstrap required, opened a new log
I20260807 02:59:23.124881 22547 ts_tablet_manager.cc:1397] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260807 02:59:23.124810 22550 log.cc:824] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:23.126544 22548 tablet_bootstrap.cc:492] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: No bootstrap required, opened a new log
I20260807 02:59:23.126832 22548 ts_tablet_manager.cc:1397] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 02:59:23.127517 22550 tablet_bootstrap.cc:492] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: No bootstrap required, opened a new log
I20260807 02:59:23.127771 22550 ts_tablet_manager.cc:1397] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260807 02:59:23.136471 22550 raft_consensus.cc:348] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:23.136989 22550 raft_consensus.cc:374] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:23.137101 22550 raft_consensus.cc:729] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f7d38fd7a4440ab6ae9d2ebac05ad9, State: Initialized, Role: FOLLOWER
I20260807 02:59:23.136970 22547 raft_consensus.cc:348] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:23.137404 22547 raft_consensus.cc:374] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:23.137501 22547 raft_consensus.cc:729] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c884fcae264409bb21cc38b26cf099, State: Initialized, Role: FOLLOWER
I20260807 02:59:23.137892 22550 consensus_queue.cc:260] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:23.138090 22547 consensus_queue.cc:260] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:23.138151 22548 raft_consensus.cc:348] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:23.138572 22548 raft_consensus.cc:374] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:23.138710 22548 raft_consensus.cc:729] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6e9893175c458cbe761e236f021cea, State: Initialized, Role: FOLLOWER
I20260807 02:59:23.139263 22320 heartbeater.cc:503] Master 127.1.121.126:33213 was elected leader, sending a full tablet report...
I20260807 02:59:23.139554 22548 consensus_queue.cc:260] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:23.139940 22550 ts_tablet_manager.cc:1428] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260807 02:59:23.141513 22529 heartbeater.cc:503] Master 127.1.121.126:33213 was elected leader, sending a full tablet report...
I20260807 02:59:23.142295 22548 ts_tablet_manager.cc:1428] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
I20260807 02:59:23.148314 22547 ts_tablet_manager.cc:1428] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Time spent starting tablet: real 0.023s user 0.007s sys 0.012s
I20260807 02:59:23.258774 22558 raft_consensus.cc:482] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:23.259066 22558 raft_consensus.cc:504] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:23.260739 22558 leader_election.cc:288] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92c884fcae264409bb21cc38b26cf099 (127.1.121.65:43631), 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
W20260807 02:59:23.267800 22530 tablet.cc:2369] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:23.268523 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.261076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.1.121.67:37747)
0807 02:59:23.261509 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.261535 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:23.261680 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:23.262092 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:23.262099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:23.262130 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:23.262417 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.262429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.263869 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.263872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.266090 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.266106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.267666 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.267672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.267924 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.267970 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 02:59:23.268078 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 02:59:23.268228 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":2}
I20260807 02:59:23.268762 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.261206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:60777 (local address 127.1.121.66:34285)
0807 02:59:23.261471 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.261479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.261930 (+ 451us) server_negotiation.cc:407] Connection header received
0807 02:59:23.262140 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.262145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.262223 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.262323 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:23.262954 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.263985 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.266429 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.267602 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.267742 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.267819 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.268383 (+ 564us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.268507 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260807 02:59:23.268532 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.261387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:37747 (local address 127.1.121.65:43631)
0807 02:59:23.261589 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.261593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.261612 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:59:23.261837 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.261841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.261936 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.262062 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:23.262560 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.263729 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.266256 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.267507 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.267685 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.267765 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.268150 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.268286 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:59:23.269287 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.261115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:34285 (local address 127.1.121.67:60777)
0807 02:59:23.261825 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.261841 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:23.261971 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:23.262366 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:23.262372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:23.262446 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:23.262811 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.262820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.264178 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.264183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.266291 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:23.266301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.267986 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.267995 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:23.268189 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.268225 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 02:59:23.268886 (+ 661us) client_negotiation.cc:240] Negotiation successful
0807 02:59:23.269024 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":657,"threads_started":1}
I20260807 02:59:23.269881 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0573bd29c10c4c1db10946993ab0f6a5" candidate_uuid: "ef6e9893175c458cbe761e236f021cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c884fcae264409bb21cc38b26cf099" is_pre_election: true
I20260807 02:59:23.270220 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0573bd29c10c4c1db10946993ab0f6a5" candidate_uuid: "ef6e9893175c458cbe761e236f021cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" is_pre_election: true
I20260807 02:59:23.270408 22031 raft_consensus.cc:2454] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6e9893175c458cbe761e236f021cea in term 0.
I20260807 02:59:23.270727 22249 raft_consensus.cc:2454] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6e9893175c458cbe761e236f021cea in term 0.
I20260807 02:59:23.271168 22351 leader_election.cc:302] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92c884fcae264409bb21cc38b26cf099, ef6e9893175c458cbe761e236f021cea; no voters:
I20260807 02:59:23.271632 22558 raft_consensus.cc:2781] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:59:23.271807 22558 raft_consensus.cc:482] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:59:23.271974 22558 raft_consensus.cc:3037] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 0 FOLLOWER]: Advancing to term 1
W20260807 02:59:23.272200 22322 tablet.cc:2369] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:23.274606 22558 raft_consensus.cc:504] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:23.275270 22558 leader_election.cc:288] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [CANDIDATE]: Term 1 election: Requested vote from peers 92c884fcae264409bb21cc38b26cf099 (127.1.121.65:43631), 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
I20260807 02:59:23.275633 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0573bd29c10c4c1db10946993ab0f6a5" candidate_uuid: "ef6e9893175c458cbe761e236f021cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c884fcae264409bb21cc38b26cf099"
I20260807 02:59:23.275759 22249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0573bd29c10c4c1db10946993ab0f6a5" candidate_uuid: "ef6e9893175c458cbe761e236f021cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9"
I20260807 02:59:23.275846 22031 raft_consensus.cc:3037] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:23.275987 22249 raft_consensus.cc:3037] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:23.279078 22031 raft_consensus.cc:2454] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6e9893175c458cbe761e236f021cea in term 1.
I20260807 02:59:23.279208 22249 raft_consensus.cc:2454] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6e9893175c458cbe761e236f021cea in term 1.
I20260807 02:59:23.279547 22351 leader_election.cc:302] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92c884fcae264409bb21cc38b26cf099, ef6e9893175c458cbe761e236f021cea; no voters:
I20260807 02:59:23.279917 22558 raft_consensus.cc:2781] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 1 FOLLOWER]: Leader election won for term 1
W20260807 02:59:23.280171 22105 tablet.cc:2369] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:23.280455 22558 raft_consensus.cc:686] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 1 LEADER]: Becoming Leader. State: Replica: ef6e9893175c458cbe761e236f021cea, State: Running, Role: LEADER
I20260807 02:59:23.281050 22558 consensus_queue.cc:237] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:23.285509 21805 catalog_manager.cc:5495] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea reported cstate change: term changed from 0 to 1, leader changed from <none> to ef6e9893175c458cbe761e236f021cea (127.1.121.67). New cstate: current_term: 1 leader_uuid: "ef6e9893175c458cbe761e236f021cea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } health_report { overall_health: UNKNOWN } } }
I20260807 02:59:23.336436 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.331125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.1.121.67:42111)
0807 02:59:23.331342 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.331350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.331373 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:59:23.331482 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.331487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.331576 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.331775 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:59:23.332142 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.332913 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.334418 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.334816 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.334906 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.335367 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:59:23.335875 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:59:23.335984 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.336068 (+ 84us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.336134 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 02:59:23.344625 22249 raft_consensus.cc:1261] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 1 FOLLOWER]: Refusing update from remote peer ef6e9893175c458cbe761e236f021cea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:23.344683 22031 raft_consensus.cc:1261] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Refusing update from remote peer ef6e9893175c458cbe761e236f021cea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:23.345670 22558 consensus_queue.cc:1035] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:23.345953 22560 consensus_queue.cc:1035] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:23.356779 22564 mvcc.cc:205] Tried to move back new op lower bound from 7315749123449835520 to 7315749123202936832. Current Snapshot: MvccSnapshot[applied={T|T < 7315749123449835520}]
I20260807 02:59:23.374763 21805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:59:23.375058 21805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:59:23.378578 21805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Sending DeleteTablet for 3 replicas of tablet 0573bd29c10c4c1db10946993ab0f6a5
I20260807 02:59:23.379900 22424 tablet_service.cc:1514] Processing DeleteTablet for tablet 0573bd29c10c4c1db10946993ab0f6a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:23 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:36864
I20260807 02:59:23.379889 22003 tablet_service.cc:1514] Processing DeleteTablet for tablet 0573bd29c10c4c1db10946993ab0f6a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:23 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:50198
I20260807 02:59:23.380144 22209 tablet_service.cc:1514] Processing DeleteTablet for tablet 0573bd29c10c4c1db10946993ab0f6a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:23 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:33986
I20260807 02:59:23.381310 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 21864
I20260807 02:59:23.409480 22579 tablet_replica.cc:330] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: stopping tablet replica
I20260807 02:59:23.411688 22579 raft_consensus.cc:2229] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:59:23.412544 22579 raft_consensus.cc:2258] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:23.410885 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.382470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.1.121.65:43631)
0807 02:59:23.382636 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.382642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.382662 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:59:23.382848 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.382857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.383014 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.383121 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:23.385425 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.386996 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.394782 (+ 7786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.395757 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.395932 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.400556 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:23.400582 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:23.400613 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:23.400686 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:23.405972 (+ 5286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.407095 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.407111 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.407117 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.407345 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.407821 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.407828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.407834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.408538 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:23.408896 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.409921 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.410260 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:59:23.414000 22098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:23.414806 22580 tablet_replica.cc:330] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: stopping tablet replica
I20260807 02:59:23.415580 22581 tablet_replica.cc:330] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: stopping tablet replica
I20260807 02:59:23.416385 22580 raft_consensus.cc:2229] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:59:23.416401 22581 raft_consensus.cc:2229] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:59:23.417311 22580 raft_consensus.cc:2258] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:23.417665 22581 raft_consensus.cc:2258] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:23.420140 22580 ts_tablet_manager.cc:1905] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:23.420574 22581 ts_tablet_manager.cc:1905] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:23.428788 22580 ts_tablet_manager.cc:1918] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:23.429201 22580 log.cc:1196] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/0573bd29c10c4c1db10946993ab0f6a5
I20260807 02:59:23.429948 22580 ts_tablet_manager.cc:1939] T 0573bd29c10c4c1db10946993ab0f6a5 P ef6e9893175c458cbe761e236f021cea: Deleting consensus metadata
I20260807 02:59:23.431978 22581 ts_tablet_manager.cc:1918] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:23.432260 22581 log.cc:1196] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/0573bd29c10c4c1db10946993ab0f6a5
I20260807 02:59:23.432241 21788 catalog_manager.cc:4841] TS ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111): tablet 0573bd29c10c4c1db10946993ab0f6a5 (table test-table [id=1f61191c4c304e589732ac9f4cd22f91]) successfully deleted
I20260807 02:59:23.432951 22581 ts_tablet_manager.cc:1939] T 0573bd29c10c4c1db10946993ab0f6a5 P 09f7d38fd7a4440ab6ae9d2ebac05ad9: Deleting consensus metadata
I20260807 02:59:23.434849 21788 catalog_manager.cc:4841] TS 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285): tablet 0573bd29c10c4c1db10946993ab0f6a5 (table test-table [id=1f61191c4c304e589732ac9f4cd22f91]) successfully deleted
I20260807 02:59:23.541426 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 21864
W20260807 02:59:23.578125 21782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.121.65:43631 (error 108)
I20260807 02:59:23.578491 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 22112
I20260807 02:59:23.580298 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.579860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50220)
0807 02:59:23.580032 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.580143 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 02:59:23.580596 21782 catalog_manager.cc:4584] TS 92c884fcae264409bb21cc38b26cf099 (127.1.121.65:43631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0573bd29c10c4c1db10946993ab0f6a5: Network error: Client connection negotiation failed: client connection to 127.1.121.65:43631: connect: Connection refused (error 111)
I20260807 02:59:23.596112 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.579521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34000 (local address 127.1.121.66:34285)
0807 02:59:23.579729 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.579735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.579756 (+ 21us) server_negotiation.cc:407] Connection header received
0807 02:59:23.580531 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.580537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.580635 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.580769 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:23.582746 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.583490 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.585511 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.585893 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.585990 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.589862 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:23.589877 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:23.589893 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:23.589931 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:23.593060 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.593952 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.593959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.593962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.594090 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.594539 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.594542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.594544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.594952 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:23.595167 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.595611 (+ 444us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.595803 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:59:23.596807 22309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:23.634552 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.633884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50228)
0807 02:59:23.634172 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.634323 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260807 02:59:23.635895 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.635423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50236)
0807 02:59:23.635618 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.635736 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 02:59:23.691771 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 22112
I20260807 02:59:23.701201 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.700471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50248)
0807 02:59:23.700767 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.700940 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260807 02:59:23.702751 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.702214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50252)
0807 02:59:23.702427 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.702566 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 02:59:23.721930 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 22324
I20260807 02:59:23.739975 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.722877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.1.121.67:42111)
0807 02:59:23.723109 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:23.723115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:23.723142 (+ 27us) server_negotiation.cc:407] Connection header received
0807 02:59:23.723357 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:23.723365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:23.723477 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:23.723607 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:23.725303 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.726746 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.728228 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:23.729099 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:23.729276 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:23.731933 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:23.731949 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:23.731971 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:23.732037 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:23.736630 (+ 4593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.737494 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.737504 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.737508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.737758 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:23.738161 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:23.738174 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:23.738178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:23.738707 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:23.739033 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:23.739218 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 02:59:23.739476 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 02:59:23.741068 22515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:23.783651 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.783110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50254)
0807 02:59:23.783362 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.783465 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260807 02:59:23.784745 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:23.784381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50270)
0807 02:59:23.784571 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:23.784642 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260807 02:59:23.871448 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 22324
I20260807 02:59:23.903900 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 21759
I20260807 02:59:23.905210 21853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:24.020514 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:24.019868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:43631 (local address 127.0.0.1:50282)
0807 02:59:24.020095 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:24.020198 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 02:59:24.020691 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 21759
I20260807 02:59:24.051080 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33213
--webserver_interface=127.1.121.126
--webserver_port=43473
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:33213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:24.421838 22593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:24.422173 22593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:24.422329 22593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:24.429692 22593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:59:24.429827 22593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:24.429919 22593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:24.430071 22593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:59:24.430241 22593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:59:24.441619 22593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:33213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:33213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=43473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:24.443135 22593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:24.444578 22593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:24.457882 22600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:24.457770 22608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:24.457827 22604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:24.458879 22593 server_base.cc:1034] running on GCE node
I20260807 02:59:24.459797 22593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:24.461408 22593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:24.462683 22593 hybrid_clock.cc:648] HybridClock initialized: now 1786071564462624 us; error 37 us; skew 500 ppm
I20260807 02:59:24.463064 22593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:24 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071564, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:24.466482 22593 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:59:24.468117 22593 webserver.cc:460] Webserver started at http://127.1.121.126:43473/ using document root <none> and password file <none>
I20260807 02:59:24.468945 22593 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:24.469115 22593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:24.474987 22593 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 02:59:24.478156 22616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:24.479518 22593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:59:24.479724 22593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "9af7655807744da8bc505e64dfd96fcc"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "7a0faf0157ca2cc200d6e16a605ea849"
server_key_iv: "68863f459141ba1a815c3cf029897204"
server_key_version: "encryptionkey@0"
I20260807 02:59:24.480870 22593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:24.495769 22593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:24.500296 22593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:24.500645 22593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:24.522132 22593 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:33213
I20260807 02:59:24.522158 22692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:33213 every 8 connection(s)
I20260807 02:59:24.523838 22593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:59:24.530893 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 22593
I20260807 02:59:24.530929 22693 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:59:24.531834 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:43631
--local_ip_for_outbound_sockets=127.1.121.65
--tserver_master_addrs=127.1.121.126:33213
--webserver_port=38401
--webserver_interface=127.1.121.65
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:59:24.534538 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Bootstrap starting.
I20260807 02:59:24.555912 22693 log.cc:824] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:24.572469 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:59:24.573179 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: Bootstrap complete.
I20260807 02:59:24.582635 22693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:24.583544 22693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af7655807744da8bc505e64dfd96fcc, State: Initialized, Role: FOLLOWER
I20260807 02:59:24.584226 22693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:24.584517 22693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:59:24.584657 22693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:59:24.584911 22693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 2 FOLLOWER]: Advancing to term 3
I20260807 02:59:24.589288 22693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:24.589839 22693 leader_election.cc:302] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9af7655807744da8bc505e64dfd96fcc; no voters:
I20260807 02:59:24.590540 22693 leader_election.cc:288] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 02:59:24.590678 22698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 3 FOLLOWER]: Leader election won for term 3
I20260807 02:59:24.592207 22698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [term 3 LEADER]: Becoming Leader. State: Replica: 9af7655807744da8bc505e64dfd96fcc, State: Running, Role: LEADER
I20260807 02:59:24.592818 22698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } }
I20260807 02:59:24.593052 22693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:59:24.595687 22700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af7655807744da8bc505e64dfd96fcc. Latest consensus state: current_term: 3 leader_uuid: "9af7655807744da8bc505e64dfd96fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } } }
I20260807 02:59:24.595732 22699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9af7655807744da8bc505e64dfd96fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af7655807744da8bc505e64dfd96fcc" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 33213 } } }
I20260807 02:59:24.596217 22700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:24.596278 22699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:24.606391 22714 catalog_manager.cc:1259] Loaded cluster ID: 4a4f2cdfd0914f2fabfc114c2b5c308b
I20260807 02:59:24.606529 22714 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: loading cluster ID for follower catalog manager: success
W20260807 02:59:24.609704 22714 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9af7655807744da8bc505e64dfd96fcc: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260807 02:59:24.609933 22714 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260807 02:59:24.610116 22707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:59:24.616964 22707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:59:24.617930 22707 catalog_manager.cc:1259] Loaded cluster ID: 4a4f2cdfd0914f2fabfc114c2b5c308b
I20260807 02:59:24.618073 22707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 02:59:24.619359 22707 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb52910f9ad google::LogMessage::Fail() at ??:0
@ 0x7fb52911288a google::LogMessage::SendToLog() at ??:0
@ 0x7fb52910f4de google::LogMessage::Flush() at ??:0
@ 0x7fb529110a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb53eb42512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb53eb3f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb53ebc7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb53ebc7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb52bd5b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb52bd5e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb52bd5e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb52bd39eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb52f0136db start_thread at ??:0
@ 0x7fb527fa961f clone at ??:0
W20260807 02:59:24.956768 22696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:24.957181 22696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:24.957379 22696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:24.965992 22696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:24.966157 22696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:24.966379 22696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:59:24.977916 22696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:43631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=38401
--tserver_master_addrs=127.1.121.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:24.979545 22696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:24.981074 22696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:24.996578 22747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:24.996688 22750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:24.996713 22748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:24.997912 22696 server_base.cc:1034] running on GCE node
I20260807 02:59:24.998554 22696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:24.999526 22696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:25.000744 22696 hybrid_clock.cc:648] HybridClock initialized: now 1786071565000672 us; error 57 us; skew 500 ppm
I20260807 02:59:25.001116 22696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:24 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071564, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:25.004455 22696 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:59:25.006050 22696 webserver.cc:460] Webserver started at http://127.1.121.65:38401/ using document root <none> and password file <none>
I20260807 02:59:25.006804 22696 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:25.006961 22696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:25.012646 22696 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 02:59:25.015836 22758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:25.017294 22696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:59:25.017488 22696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "92c884fcae264409bb21cc38b26cf099"
format_stamp: "Formatted at 2026-08-07 02:59:18 on dist-test-slave-3h5h"
server_key: "1552f0a90a0fc7df064f8e6096fad3b3"
server_key_iv: "edd599a26292141c8bfd44d7d970fa2f"
server_key_version: "encryptionkey@0"
I20260807 02:59:25.018504 22696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:25.034432 22696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:25.039171 22696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:25.039498 22696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:25.040568 22696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:25.042743 22776 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 02:59:25.049350 22696 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 02:59:25.049503 22696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260807 02:59:25.049649 22696 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 02:59:25.052901 22696 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 02:59:25.053013 22696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.003s
I20260807 02:59:25.053416 22776 tablet_bootstrap.cc:492] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Bootstrap starting.
I20260807 02:59:25.064173 22776 log.cc:824] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:25.083485 22776 tablet_bootstrap.cc:492] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:59:25.084740 22776 tablet_bootstrap.cc:492] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Bootstrap complete.
I20260807 02:59:25.085631 22776 ts_tablet_manager.cc:1397] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.008s
I20260807 02:59:25.096284 22776 raft_consensus.cc:348] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:25.097791 22776 raft_consensus.cc:729] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c884fcae264409bb21cc38b26cf099, State: Initialized, Role: FOLLOWER
I20260807 02:59:25.098702 22776 consensus_queue.cc:260] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:25.100629 22776 ts_tablet_manager.cc:1428] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260807 02:59:25.106611 22696 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:43631
I20260807 02:59:25.106648 22952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:43631 every 8 connection(s)
I20260807 02:59:25.108134 22696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:59:25.113566 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:25.111592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:41389)
0807 02:59:25.112402 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:25.112633 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":342,"threads_started":1}
W20260807 02:59:25.114265 22953 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:33213 (0 consecutive failures): Network error: Failed to ping master at 127.1.121.126:33213: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
I20260807 02:59:25.115115 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:25.114804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:43525)
0807 02:59:25.114947 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:25.115013 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 02:59:25.115972 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:25.115667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:43711)
0807 02:59:25.115797 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:25.115863 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 02:59:25.116235 22953 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:59:25.116907 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 22696
I20260807 02:59:25.117533 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:34285
--local_ip_for_outbound_sockets=127.1.121.66
--tserver_master_addrs=127.1.121.126:33213
--webserver_port=36623
--webserver_interface=127.1.121.66
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:25.477502 22957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:25.477788 22957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:25.477986 22957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:25.485414 22957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:25.485548 22957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:25.485730 22957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:59:25.496096 22957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:34285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=36623
--tserver_master_addrs=127.1.121.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:25.497617 22957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:25.498999 22957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:25.513216 22969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:25.513321 22966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:25.513814 22973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:25.514129 22957 server_base.cc:1034] running on GCE node
I20260807 02:59:25.514776 22957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:25.515805 22957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:25.517000 22957 hybrid_clock.cc:648] HybridClock initialized: now 1786071565516950 us; error 51 us; skew 500 ppm
I20260807 02:59:25.517343 22957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:25 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071565, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:25.520512 22957 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:59:25.522017 22957 webserver.cc:460] Webserver started at http://127.1.121.66:36623/ using document root <none> and password file <none>
I20260807 02:59:25.522792 22957 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:25.522897 22957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:25.528260 22957 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:59:25.530931 22982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:25.532248 22957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 02:59:25.532384 22957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "fc41626422d40bef2ce894a7b8be1b78"
server_key_iv: "c724addfbbe8cd5ba20db2ec51c29a9a"
server_key_version: "encryptionkey@0"
I20260807 02:59:25.533398 22957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:25.560070 22957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:25.564086 22957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:25.564334 22957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:25.565325 22957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:25.566617 22957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:25.566711 22957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:25.566850 22957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:25.566928 22957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:25.601197 22957 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:34285
I20260807 02:59:25.601226 23150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:34285 every 8 connection(s)
I20260807 02:59:25.602775 22957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:59:25.607746 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:25.605845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:44317)
0807 02:59:25.606572 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:25.606887 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":270,"threads_started":1}
W20260807 02:59:25.608455 23152 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:33213 (0 consecutive failures): Network error: Failed to ping master at 127.1.121.126:33213: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
I20260807 02:59:25.609345 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:25.609034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:34073)
0807 02:59:25.609171 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:25.609241 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 02:59:25.610227 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:25.609876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:44343)
0807 02:59:25.610014 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:25.610104 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 02:59:25.610497 23152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:59:25.611070 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 22957
I20260807 02:59:25.611840 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:42111
--local_ip_for_outbound_sockets=127.1.121.67
--tserver_master_addrs=127.1.121.126:33213
--webserver_port=37291
--webserver_interface=127.1.121.67
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:25.966784 23155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:25.967161 23155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:25.967347 23155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:25.974779 23155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:25.974954 23155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:25.975155 23155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:59:25.985076 23155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:42111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=37291
--tserver_master_addrs=127.1.121.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:25.986464 23155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:25.987856 23155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:26.002822 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:26.002843 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:26.002957 23169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:26.003619 23155 server_base.cc:1034] running on GCE node
I20260807 02:59:26.004292 23155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:26.005378 23155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:26.006633 23155 hybrid_clock.cc:648] HybridClock initialized: now 1786071566006580 us; error 45 us; skew 500 ppm
I20260807 02:59:26.006986 23155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:26 dist-test-slave-3h5h krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071566, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:26.010574 23155 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:59:26.012310 23155 webserver.cc:460] Webserver started at http://127.1.121.67:37291/ using document root <none> and password file <none>
I20260807 02:59:26.013211 23155 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:26.013588 23155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:26.019536 23155 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260807 02:59:26.022768 23180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:26.024273 23155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:59:26.024457 23155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "ef6e9893175c458cbe761e236f021cea"
format_stamp: "Formatted at 2026-08-07 02:59:19 on dist-test-slave-3h5h"
server_key: "0c21ef68e6ac375be49653319d8a6d23"
server_key_iv: "28fc13ea429f1eeb0664d120a5daa5f6"
server_key_version: "encryptionkey@0"
I20260807 02:59:26.025687 23155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:26.056175 23155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:26.060566 23155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:26.060967 23155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:26.062029 23155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:26.063423 23155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:26.063553 23155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:26.063691 23155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:26.063781 23155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:26.100485 23155 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:42111
I20260807 02:59:26.100543 23351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:42111 every 8 connection(s)
I20260807 02:59:26.102149 23155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:59:26.103482 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 23155
I20260807 02:59:26.108135 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:26.105953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:39333)
0807 02:59:26.106851 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:26.107218 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":370,"threads_started":1}
W20260807 02:59:26.108968 23352 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:33213 (0 consecutive failures): Network error: Failed to ping master at 127.1.121.126:33213: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
I20260807 02:59:26.109892 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:26.109562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:47541)
0807 02:59:26.109714 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:26.109787 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 02:59:26.110842 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:26.110488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:37287)
0807 02:59:26.110644 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:26.110728 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 02:59:26.111104 23352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:59:26.117657 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:26.116872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:58313)
0807 02:59:26.117332 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:26.117430 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260807 02:59:26.611894 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:26.611096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:52473)
0807 02:59:26.611562 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:26.611654 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260807 02:59:26.854668 23363 raft_consensus.cc:482] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:26.854991 23363 raft_consensus.cc:504] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:26.856516 23363 leader_election.cc:288] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111), 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
Aug 07 02:59:26 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071564, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
Aug 07 02:59:26 dist-test-slave-3h5h krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071564, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:59:26.883597 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:26.856851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:42111 (local address 127.1.121.65:55045)
0807 02:59:26.857454 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:26.857500 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:26.858127 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:26.861827 (+ 3700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:26.861840 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:26.862825 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:26.863454 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:26.863474 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:26.865098 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:26.865102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:26.866453 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:26.866471 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:26.867487 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:26.867493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:26.867799 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:26.873109 (+ 5310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:26.873164 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:26.876364 (+ 3200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:26.879970 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:26.879980 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:26.879996 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:26.880433 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:26.880926 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:26.880932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:26.880935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:26.881119 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:26.881943 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:26.881951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:26.882252 (+ 301us) client_negotiation.cc:769] Sending connection context
0807 02:59:26.882523 (+ 271us) client_negotiation.cc:240] Negotiation successful
0807 02:59:26.883120 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":94,"spinlock_wait_cycles":11776,"thread_start_us":292,"threads_started":1}
I20260807 02:59:26.884024 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:26.857064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:55045 (local address 127.1.121.67:42111)
0807 02:59:26.859264 (+ 2200us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:26.859276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:26.859308 (+ 32us) server_negotiation.cc:407] Connection header received
0807 02:59:26.860812 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:26.860825 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:26.861292 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:26.861670 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:26.863658 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:26.864883 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:26.866609 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:26.867249 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:26.867458 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:26.876717 (+ 9259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:26.876726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:26.876741 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:26.876782 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:26.879812 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:26.880597 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:26.880604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:26.880609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:26.880775 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:26.881236 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:26.881244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:26.881247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:26.881783 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:26.881985 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:26.882791 (+ 806us) server_negotiation.cc:299] Negotiation successful
0807 02:59:26.883400 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2007,"thread_start_us":1900,"threads_started":1}
I20260807 02:59:26.884029 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:26.857578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:34285 (local address 127.1.121.65:52935)
0807 02:59:26.858971 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:26.858991 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:26.859102 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:26.862301 (+ 3199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:26.862304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:26.862858 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:26.863457 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:26.863475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:26.865100 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:26.865104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:26.866456 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:26.866472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:26.867423 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:26.867430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:26.867806 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:26.873109 (+ 5303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:26.873153 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:26.876555 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:26.880261 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:26.880268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:26.880272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:26.880659 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:26.881179 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:26.881184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:26.881187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:26.881333 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:26.882094 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:26.882099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:26.882286 (+ 187us) client_negotiation.cc:769] Sending connection context
0807 02:59:26.882522 (+ 236us) client_negotiation.cc:240] Negotiation successful
0807 02:59:26.883389 (+ 867us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"mutex_wait_us":425,"spinlock_wait_cycles":42624,"thread_start_us":1079,"threads_started":1}
I20260807 02:59:26.885607 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:26.858062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:52935 (local address 127.1.121.66:34285)
0807 02:59:26.859651 (+ 1589us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:26.859662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:26.859698 (+ 36us) server_negotiation.cc:407] Connection header received
0807 02:59:26.861450 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:26.861462 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:26.861895 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:26.862239 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:26.863677 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:26.864907 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:26.866606 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:26.867217 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:26.867376 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:26.876911 (+ 9535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:26.876921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:26.876935 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:26.876976 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:26.880099 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:26.880821 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:26.880828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:26.880880 (+ 52us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:26.881043 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:26.881450 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:26.881455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:26.881458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:26.881949 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:26.882148 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:26.884374 (+ 2226us) server_negotiation.cc:299] Negotiation successful
0807 02:59:26.885225 (+ 851us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1438,"thread_start_us":1268,"threads_started":1}
W20260807 02:59:26.887727 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
W20260807 02:59:26.888239 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
I20260807 02:59:26.888401 22764 leader_election.cc:302] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92c884fcae264409bb21cc38b26cf099; no voters: 09f7d38fd7a4440ab6ae9d2ebac05ad9, ef6e9893175c458cbe761e236f021cea
I20260807 02:59:26.888890 23363 raft_consensus.cc:2726] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:59:27.112414 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:27.111591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:59141)
0807 02:59:27.112034 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:27.112180 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":210,"threads_started":1}
I20260807 02:59:27.118957 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:27.118539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:59015)
0807 02:59:27.118725 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:27.118827 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 02:59:27.613471 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:27.612756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:57969)
0807 02:59:27.613229 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:27.613311 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":273,"threads_started":1}
I20260807 02:59:28.114248 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:28.113364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:39091)
0807 02:59:28.113898 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:28.114048 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":218,"threads_started":1}
I20260807 02:59:28.120570 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:28.119842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:36079)
0807 02:59:28.120322 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:28.120410 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":250,"threads_started":1}
I20260807 02:59:28.621553 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:28.614965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:36677)
0807 02:59:28.621269 (+ 6304us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:28.621364 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6174,"thread_start_us":6068,"threads_started":1}
I20260807 02:59:28.929147 23379 raft_consensus.cc:482] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:28.929383 23379 raft_consensus.cc:504] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:28.930014 23379 leader_election.cc:288] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111), 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
W20260807 02:59:28.931077 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
W20260807 02:59:28.931340 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
I20260807 02:59:28.931456 22764 leader_election.cc:302] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92c884fcae264409bb21cc38b26cf099; no voters: 09f7d38fd7a4440ab6ae9d2ebac05ad9, ef6e9893175c458cbe761e236f021cea
I20260807 02:59:28.931721 23379 raft_consensus.cc:2726] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:59:29.115754 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:29.115066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:40033)
0807 02:59:29.115515 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:29.115596 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":240,"threads_started":1}
I20260807 02:59:29.122501 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:29.121534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:55807)
0807 02:59:29.122127 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:29.122267 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":321,"threads_started":1}
I20260807 02:59:29.623335 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:29.622512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:35259)
0807 02:59:29.623054 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:29.623147 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":267,"threads_started":1}
I20260807 02:59:30.117795 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:30.116759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:59251)
0807 02:59:30.117522 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:30.117609 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":557,"threads_started":1}
I20260807 02:59:30.124392 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:30.123505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:52901)
0807 02:59:30.124094 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:30.124199 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":313,"threads_started":1}
I20260807 02:59:30.532974 23387 raft_consensus.cc:482] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:30.533217 23387 raft_consensus.cc:504] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:30.533943 23387 leader_election.cc:288] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111), 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
W20260807 02:59:30.534931 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
W20260807 02:59:30.535143 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
I20260807 02:59:30.535269 22764 leader_election.cc:302] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92c884fcae264409bb21cc38b26cf099; no voters: 09f7d38fd7a4440ab6ae9d2ebac05ad9, ef6e9893175c458cbe761e236f021cea
I20260807 02:59:30.535542 23387 raft_consensus.cc:2726] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:59:30.625336 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:30.624366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:51533)
0807 02:59:30.624934 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:30.625058 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":297,"threads_started":1}
I20260807 02:59:31.119622 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:31.118842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:42085)
0807 02:59:31.119339 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:31.119437 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":243,"threads_started":1}
I20260807 02:59:31.126109 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:31.125352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:55377)
0807 02:59:31.125853 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:31.125945 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":295,"threads_started":1}
I20260807 02:59:31.627158 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:31.626310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:57533)
0807 02:59:31.626889 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:31.626972 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":332,"threads_started":1}
I20260807 02:59:32.121480 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:32.120555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:59769)
0807 02:59:32.121218 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:32.121305 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":356,"threads_started":1}
I20260807 02:59:32.127873 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:32.127073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:52667)
0807 02:59:32.127578 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:32.127671 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":288,"threads_started":1}
I20260807 02:59:32.629076 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:32.628194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:59785)
0807 02:59:32.628691 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:32.628791 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":208,"threads_started":1}
I20260807 02:59:33.123531 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:33.122417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:50027)
0807 02:59:33.123231 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:33.123341 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":370,"threads_started":1}
I20260807 02:59:33.129675 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:33.128660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:58443)
0807 02:59:33.129242 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:33.129337 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":337,"threads_started":1}
I20260807 02:59:33.630853 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:33.630045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:42499)
0807 02:59:33.630564 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:33.630673 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":299,"threads_started":1}
I20260807 02:59:34.125538 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:34.124598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:36649)
0807 02:59:34.125188 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:34.125299 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":295,"threads_started":1}
I20260807 02:59:34.131242 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:34.130511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:36049)
0807 02:59:34.130948 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:34.131028 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":216,"threads_started":1}
I20260807 02:59:34.632599 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:34.631867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:58653)
0807 02:59:34.632346 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:34.632436 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":219,"threads_started":1}
I20260807 02:59:34.910869 23407 raft_consensus.cc:482] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:34.911098 23407 raft_consensus.cc:504] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:34.911721 23407 leader_election.cc:288] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111), 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
W20260807 02:59:34.912719 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
W20260807 02:59:34.912979 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
I20260807 02:59:34.913070 22764 leader_election.cc:302] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92c884fcae264409bb21cc38b26cf099; no voters: 09f7d38fd7a4440ab6ae9d2ebac05ad9, ef6e9893175c458cbe761e236f021cea
I20260807 02:59:34.913359 23407 raft_consensus.cc:2726] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:59:35.127169 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:35.126441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:46111)
0807 02:59:35.126930 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:35.127013 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260807 02:59:35.133122 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:35.132207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:46653)
0807 02:59:35.132779 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:35.132929 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":308,"threads_started":1}
I20260807 02:59:35.634413 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:35.633601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:49621)
0807 02:59:35.634130 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:35.634226 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":290,"threads_started":1}
I20260807 02:59:36.129071 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:36.128149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:48909)
0807 02:59:36.128723 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:36.128822 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":343,"threads_started":1}
I20260807 02:59:36.134711 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:36.134020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:56511)
0807 02:59:36.134462 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:36.134541 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":240,"threads_started":1}
I20260807 02:59:36.636089 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:36.635391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:33785)
0807 02:59:36.635835 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:36.635918 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":208,"threads_started":1}
I20260807 02:59:37.131016 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:37.130194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:55589)
0807 02:59:37.130760 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:37.130858 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":273,"threads_started":1}
I20260807 02:59:37.136747 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:37.135629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:35591)
0807 02:59:37.136308 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:37.136445 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":293,"threads_started":1}
I20260807 02:59:37.637894 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:37.637129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:35291)
0807 02:59:37.637600 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:37.637712 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":204,"threads_started":1}
I20260807 02:59:37.908730 23419 raft_consensus.cc:482] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:37.909005 23419 raft_consensus.cc:504] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef6e9893175c458cbe761e236f021cea" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 42111 } } peers { permanent_uuid: "92c884fcae264409bb21cc38b26cf099" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 43631 } } peers { permanent_uuid: "09f7d38fd7a4440ab6ae9d2ebac05ad9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 34285 } }
I20260807 02:59:37.909674 23419 leader_election.cc:288] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111), 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285)
W20260807 02:59:37.910667 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef6e9893175c458cbe761e236f021cea (127.1.121.67:42111): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
W20260807 02:59:37.910881 22764 leader_election.cc:341] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f7d38fd7a4440ab6ae9d2ebac05ad9 (127.1.121.66:34285): Not found: Tablet not found: 0573bd29c10c4c1db10946993ab0f6a5
I20260807 02:59:37.910970 22764 leader_election.cc:302] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92c884fcae264409bb21cc38b26cf099; no voters: 09f7d38fd7a4440ab6ae9d2ebac05ad9, ef6e9893175c458cbe761e236f021cea
I20260807 02:59:37.911206 23419 raft_consensus.cc:2726] T 0573bd29c10c4c1db10946993ab0f6a5 P 92c884fcae264409bb21cc38b26cf099 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:59:38.133020 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:38.132069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:51605)
0807 02:59:38.132679 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:38.132782 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":336,"threads_started":1}
I20260807 02:59:38.138753 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:38.137748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:57099)
0807 02:59:38.138395 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:38.138527 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":375,"threads_started":1}
I20260807 02:59:38.639839 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:38.638875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:50611)
0807 02:59:38.639514 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:38.639653 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":333,"threads_started":1}
I20260807 02:59:39.134703 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:39.133967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:56587)
0807 02:59:39.134408 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:39.134513 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":225,"threads_started":1}
I20260807 02:59:39.140560 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:39.139659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:57221)
0807 02:59:39.140230 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:39.140341 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":386,"threads_started":1}
I20260807 02:59:39.641624 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:39.640742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:54485)
0807 02:59:39.641349 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:39.641432 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":363,"threads_started":1}
I20260807 02:59:40.136435 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:40.135620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:45943)
0807 02:59:40.136083 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:40.136217 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260807 02:59:40.142863 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:40.141934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.65:59189)
0807 02:59:40.142580 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:40.142666 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":376,"threads_started":1}
I20260807 02:59:40.643864 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:40.642832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.66:36587)
0807 02:59:40.643445 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:40.643603 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260807 02:59:41.107861 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 22696
I20260807 02:59:41.124352 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:41.109200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.1.121.65:43631)
0807 02:59:41.110296 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:41.110310 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:41.110348 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:59:41.110433 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:41.110445 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:41.110802 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:41.111013 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:41.112515 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:41.113596 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:41.114875 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:41.115422 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:41.115566 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:41.118461 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:41.118473 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:41.118487 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:41.118521 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:41.121394 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:41.122178 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:41.122192 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:41.122194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:41.122331 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:41.122720 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:41.122725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:41.122728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:41.123119 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:41.123438 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:41.123758 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 02:59:41.124023 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":720,"threads_started":1}
I20260807 02:59:41.125661 22948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:41.138641 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:41.137833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:33213 (local address 127.1.121.67:47025)
0807 02:59:41.138370 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:41.138467 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":288,"threads_started":1}
I20260807 02:59:41.216559 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 22696
I20260807 02:59:41.245327 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 22957
I20260807 02:59:41.259243 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:41.246298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.1.121.66:34285)
0807 02:59:41.246856 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:41.246861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:41.246878 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:59:41.246936 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:41.246939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:41.247000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:41.247096 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:41.248502 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:41.249458 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:41.250666 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:41.251112 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:41.251204 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:41.253993 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:41.253998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:41.254002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:41.254034 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:41.256811 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:41.257489 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:41.257493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:41.257496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:41.257621 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:41.257950 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:41.257954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:41.257955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:41.258221 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:41.258417 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:41.258648 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 02:59:41.258832 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":308,"threads_started":1}
I20260807 02:59:41.259999 23146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:41.347693 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 22957
I20260807 02:59:41.374115 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 23155
I20260807 02:59:41.387789 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:41.375141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.1.121.67:42111)
0807 02:59:41.375775 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:41.375780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:41.375797 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:59:41.375854 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:41.375858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:41.375919 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:41.376014 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:41.377442 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:41.378279 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:41.379507 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:41.379906 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:41.380000 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:41.382683 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:41.382687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:41.382691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:41.382722 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:41.385423 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:41.386087 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:41.386092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:41.386094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:41.386250 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:41.386575 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:41.386578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:41.386580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:41.386843 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:41.387040 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:41.387290 (+ 250us) server_negotiation.cc:299] Negotiation successful
0807 02:59:41.387459 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":317,"threads_started":1}
I20260807 02:59:41.388569 23347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:41.476195 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 23155
2026-08-07T02:59:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25428 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:59:41 dist-test-slave-3h5h krb5kdc[23456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:59:41 dist-test-slave-3h5h krb5kdc[23456](info): set up 2 sockets
Aug 07 02:59:41 dist-test-slave-3h5h krb5kdc[23456](info): commencing operation
krb5kdc: starting...
W20260807 02:59:42.794935 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 02:59:43 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 02:59:43.124696 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37333
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:37333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:43.491358 23478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:43.491649 23478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:43.491739 23478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:43.499015 23478 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 02:59:43.499128 23478 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 02:59:43.499168 23478 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 02:59:43.499223 23478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:59:43.499279 23478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:43.499356 23478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:43.499404 23478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:59:43.499454 23478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:59:43.509471 23478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:37333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:43.510751 23478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:43.512048 23478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:43.524358 23491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:43.524396 23489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:43.524433 23488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:43.525359 23478 server_base.cc:1034] running on GCE node
I20260807 02:59:43.526121 23478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:43.527462 23478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:43.528693 23478 hybrid_clock.cc:648] HybridClock initialized: now 1786071583528639 us; error 24 us; skew 500 ppm
I20260807 02:59:43.529059 23478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:43 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:43.532146 23478 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 02:59:43.533707 23478 webserver.cc:460] Webserver started at http://127.1.121.126:34567/ using document root <none> and password file <none>
I20260807 02:59:43.534392 23478 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:43.534478 23478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:43.534746 23478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:43.537818 23478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "cde83b0ae3e74ce8ae4cbb38103455ec"
format_stamp: "Formatted at 2026-08-07 02:59:43 on dist-test-slave-3h5h"
server_key: "95e59efdf7d598d2defd977309d47a9c"
server_key_iv: "7c94d60409cb44a7500c28a85845d7eb"
server_key_version: "encryptionkey@0"
I20260807 02:59:43.538586 23478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "cde83b0ae3e74ce8ae4cbb38103455ec"
format_stamp: "Formatted at 2026-08-07 02:59:43 on dist-test-slave-3h5h"
server_key: "95e59efdf7d598d2defd977309d47a9c"
server_key_iv: "7c94d60409cb44a7500c28a85845d7eb"
server_key_version: "encryptionkey@0"
I20260807 02:59:43.543673 23478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 02:59:43.547199 23501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:43.548521 23478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 02:59:43.548667 23478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "cde83b0ae3e74ce8ae4cbb38103455ec"
format_stamp: "Formatted at 2026-08-07 02:59:43 on dist-test-slave-3h5h"
server_key: "95e59efdf7d598d2defd977309d47a9c"
server_key_iv: "7c94d60409cb44a7500c28a85845d7eb"
server_key_version: "encryptionkey@0"
I20260807 02:59:43.548893 23478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:43.593271 23478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:43.597373 23478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:43.597599 23478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:43.617152 23478 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:37333
I20260807 02:59:43.617159 23578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:37333 every 8 connection(s)
I20260807 02:59:43.618574 23478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 02:59:43.622457 23579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:43.628727 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 23478
I20260807 02:59:43.629024 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 02:59:43.629698 1509 external_mini_cluster.cc:1428] Setting key bfcfb4d7ddffb2f8f4d7bd5923fe50b6
I20260807 02:59:43.633056 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Bootstrap starting.
I20260807 02:59:43.636606 23579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:43.637830 23579 log.cc:824] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:43.641185 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: No bootstrap required, opened a new log
Aug 07 02:59:43 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:43.649262 23579 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 02:59:43.649655 23579 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:43.649813 23579 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde83b0ae3e74ce8ae4cbb38103455ec, State: Initialized, Role: FOLLOWER
I20260807 02:59:43.650538 23579 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 02:59:43.650817 23579 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:59:43.650977 23579 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:59:43.651175 23579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:43.653841 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:43.631704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.1.121.126:37333)
0807 02:59:43.632478 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:43.632490 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:43.632531 (+ 41us) server_negotiation.cc:407] Connection header received
0807 02:59:43.634335 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:43.634358 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:43.634728 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:43.635042 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:43.637053 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:43.638580 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:43.640495 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:43.641215 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:43.641424 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:43.645623 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:43.645631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:43.645648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:43.645683 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:43.648958 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:43.649639 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:43.649649 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:43.649664 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:43.649877 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:43.650390 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:43.650394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:43.650396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:43.651018 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:43.651350 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:43.651962 (+ 612us) server_negotiation.cc:299] Negotiation successful
0807 02:59:43.652666 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":372,"threads_started":1}
I20260807 02:59:43.653997 23579 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 02:59:43.654546 23579 leader_election.cc:302] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde83b0ae3e74ce8ae4cbb38103455ec; no voters:
I20260807 02:59:43.655244 23579 leader_election.cc:288] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:59:43.655362 23585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:59:43.657022 23585 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 1 LEADER]: Becoming Leader. State: Replica: cde83b0ae3e74ce8ae4cbb38103455ec, State: Running, Role: LEADER
I20260807 02:59:43.657841 23579 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:59:43.657696 23585 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 02:59:43.661023 23589 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } } }
I20260807 02:59:43.661230 23590 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde83b0ae3e74ce8ae4cbb38103455ec. Latest consensus state: current_term: 1 leader_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } } }
I20260807 02:59:43.661562 23589 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:43.661695 23590 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: This master's current role is: LEADER
I20260807 02:59:43.663515 23601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:59:43.668710 23601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:59:43.678308 23601 catalog_manager.cc:1347] Generated new cluster ID: 69c6c09a3eb644d9a72f86f7a2cee67c
I20260807 02:59:43.678429 23601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:59:43.703832 23601 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:59:43.705267 23601 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:59:43.719326 23601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Generated new TSK 0
I20260807 02:59:43.720420 23601 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 02:59:43.796802 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37333
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:44.163245 23620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:44.163661 23620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:44.163856 23620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:44.172364 23620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:44.172547 23620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:44.172751 23620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 02:59:44.184267 23620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:44.185830 23620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:44.187330 23620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:44.201886 23629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:44.201987 23632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:44.202591 23627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:44.202955 23620 server_base.cc:1034] running on GCE node
I20260807 02:59:44.203660 23620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:44.204651 23620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:44.206060 23620 hybrid_clock.cc:648] HybridClock initialized: now 1786071584206013 us; error 55 us; skew 500 ppm
I20260807 02:59:44.206437 23620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:44 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071584, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:44.213366 23620 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 02:59:44.214928 23620 webserver.cc:460] Webserver started at http://127.1.121.65:42421/ using document root <none> and password file <none>
I20260807 02:59:44.215687 23620 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:44.215848 23620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:44.216208 23620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:44.219517 23620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "5221b080598044aea73202497087d5cf"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "2321c16d41a71e9a4d0c228604e71216"
server_key_iv: "a5f423b03e70a38988d4c0b5a59a8ba2"
server_key_version: "encryptionkey@0"
I20260807 02:59:44.220363 23620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "5221b080598044aea73202497087d5cf"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "2321c16d41a71e9a4d0c228604e71216"
server_key_iv: "a5f423b03e70a38988d4c0b5a59a8ba2"
server_key_version: "encryptionkey@0"
I20260807 02:59:44.225916 23620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 02:59:44.229573 23643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:44.230916 23620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:59:44.231104 23620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "5221b080598044aea73202497087d5cf"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "2321c16d41a71e9a4d0c228604e71216"
server_key_iv: "a5f423b03e70a38988d4c0b5a59a8ba2"
server_key_version: "encryptionkey@0"
I20260807 02:59:44.231313 23620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:44.247756 23620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:44.251868 23620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:44.252238 23620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:44.253396 23620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:44.254822 23620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:44.254930 23620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:44.255088 23620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:44.255162 23620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:44.289587 23620 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:39757
I20260807 02:59:44.289614 23814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:39757 every 8 connection(s)
I20260807 02:59:44.290970 23620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 02:59:44.300084 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 23620
I20260807 02:59:44.300324 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 02:59:44.301221 1509 external_mini_cluster.cc:1428] Setting key 090beb476b8d34b0672608ac2ecd383c
Aug 07 02:59:44 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071584, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:44.313722 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:44.293561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:44257 (local address 127.1.121.126:37333)
0807 02:59:44.294092 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:44.294097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:44.294821 (+ 724us) server_negotiation.cc:407] Connection header received
0807 02:59:44.296759 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:44.296764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:44.296920 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:44.297049 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:44.298546 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:44.299298 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:44.301755 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:44.302334 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:44.302489 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:44.306187 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:44.306196 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:44.306203 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:44.306250 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:44.310222 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:44.310985 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:44.310994 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:44.311000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:44.311151 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:44.311615 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:44.311623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:44.311629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:44.312023 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:44.312190 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:44.312960 (+ 770us) server_negotiation.cc:299] Negotiation successful
0807 02:59:44.313221 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
I20260807 02:59:44.314512 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:44.293943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:44257)
0807 02:59:44.294661 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:44.294700 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:44.296487 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:44.297291 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:44.297298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:44.297794 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:44.298364 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:44.298381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:44.299481 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:44.299485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:44.301606 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:44.301622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:44.302573 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:44.302582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:44.302863 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:44.303623 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:44.303647 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:44.305948 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:44.310387 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:44.310394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:44.310408 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:44.310857 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:44.311269 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:44.311272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:44.311274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:44.311473 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:44.312187 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:44.312193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:44.312532 (+ 339us) client_negotiation.cc:769] Sending connection context
0807 02:59:44.312792 (+ 260us) client_negotiation.cc:240] Negotiation successful
0807 02:59:44.313300 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":258,"threads_started":1}
I20260807 02:59:44.316228 23815 heartbeater.cc:347] Connected to a master server at 127.1.121.126:37333
I20260807 02:59:44.316546 23815 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:44.317408 23815 heartbeater.cc:511] Master 127.1.121.126:37333 requested a full tablet report, sending...
I20260807 02:59:44.319780 23526 ts_manager.cc:194] Registered new tserver with Master: 5221b080598044aea73202497087d5cf (127.1.121.65:39757)
I20260807 02:59:44.322114 23526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:44257
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 02:59:44.371838 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37333
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:44.721798 23823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:44.722076 23823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:44.722280 23823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:44.729730 23823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:44.729880 23823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:44.730072 23823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 02:59:44.740276 23823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:44.741750 23823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:44.743152 23823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:59:44.757232 23828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:44.757258 23831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:44.757287 23834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:44.758255 23823 server_base.cc:1034] running on GCE node
I20260807 02:59:44.758849 23823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:44.759845 23823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:44.761049 23823 hybrid_clock.cc:648] HybridClock initialized: now 1786071584760995 us; error 48 us; skew 500 ppm
I20260807 02:59:44.761343 23823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:44 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071584, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:44.764540 23823 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 02:59:44.766147 23823 webserver.cc:460] Webserver started at http://127.1.121.66:45163/ using document root <none> and password file <none>
I20260807 02:59:44.766822 23823 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:44.766932 23823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:44.767274 23823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:44.770377 23823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "abd3f1f186fb4d62b4374ab35e776d4b"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "bc35f11efd75c19ed823253c9a6267f9"
server_key_iv: "f6a20625006c1316b056bb51c3bd302d"
server_key_version: "encryptionkey@0"
I20260807 02:59:44.771157 23823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "abd3f1f186fb4d62b4374ab35e776d4b"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "bc35f11efd75c19ed823253c9a6267f9"
server_key_iv: "f6a20625006c1316b056bb51c3bd302d"
server_key_version: "encryptionkey@0"
I20260807 02:59:44.776732 23823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 02:59:44.780500 23845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:44.782128 23823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:59:44.782337 23823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "abd3f1f186fb4d62b4374ab35e776d4b"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "bc35f11efd75c19ed823253c9a6267f9"
server_key_iv: "f6a20625006c1316b056bb51c3bd302d"
server_key_version: "encryptionkey@0"
I20260807 02:59:44.782564 23823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:44.809234 23823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:44.814096 23823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:44.814437 23823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:44.815673 23823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:44.817162 23823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:44.817278 23823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:44.817428 23823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:44.817500 23823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:44.852825 23823 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:39309
I20260807 02:59:44.852875 24019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:39309 every 8 connection(s)
I20260807 02:59:44.854346 23823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 02:59:44.862874 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 23823
I20260807 02:59:44.863076 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 02:59:44.863629 1509 external_mini_cluster.cc:1428] Setting key 961fdb34d75febb4f2090f16b0484dd3
Aug 07 02:59:44 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071584, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:44.875871 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:44.856976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:60099 (local address 127.1.121.126:37333)
0807 02:59:44.857480 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:44.857485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:44.858326 (+ 841us) server_negotiation.cc:407] Connection header received
0807 02:59:44.860297 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:44.860303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:44.860399 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:44.860525 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:44.862411 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:44.863555 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:44.865404 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:44.865888 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:44.866035 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:44.869940 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:44.869944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:44.869948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:44.869980 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:44.872532 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:44.873397 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:44.873401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:44.873404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:44.873541 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:44.874011 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:44.874023 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:44.874026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:44.874297 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:44.874480 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:44.875207 (+ 727us) server_negotiation.cc:299] Negotiation successful
0807 02:59:44.875459 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":284,"threads_started":1}
I20260807 02:59:44.877039 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:44.857383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:60099)
0807 02:59:44.858121 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:44.858204 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:44.859994 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:44.860783 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:44.860791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:44.861379 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:44.862206 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:44.862237 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:44.863752 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:44.863766 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:44.865222 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:44.865241 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:44.866131 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:44.866139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:44.866379 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:44.867195 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:44.867219 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:44.869680 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:44.872689 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:44.872698 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:44.872716 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:44.873250 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:44.873661 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:44.873667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:44.873670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:44.873880 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:44.874440 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:44.874449 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:44.874824 (+ 375us) client_negotiation.cc:769] Sending connection context
0807 02:59:44.875103 (+ 279us) client_negotiation.cc:240] Negotiation successful
0807 02:59:44.875686 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":282,"threads_started":1}
I20260807 02:59:44.878754 24020 heartbeater.cc:347] Connected to a master server at 127.1.121.126:37333
I20260807 02:59:44.879091 24020 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:44.879827 24020 heartbeater.cc:511] Master 127.1.121.126:37333 requested a full tablet report, sending...
I20260807 02:59:44.881424 23526 ts_manager.cc:194] Registered new tserver with Master: abd3f1f186fb4d62b4374ab35e776d4b (127.1.121.66:39309)
I20260807 02:59:44.882251 23526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:60099
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 02:59:44.936304 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37333
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:59:45.300282 24031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:59:45.300592 24031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:59:45.300771 24031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:59:45.309885 24031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:59:45.310014 24031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:59:45.310189 24031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 02:59:45.320329 24031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 02:59:45.321729 24031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:59:45.323118 24031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:59:45.325268 23815 heartbeater.cc:503] Master 127.1.121.126:37333 was elected leader, sending a full tablet report...
W20260807 02:59:45.337862 24039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:45.337929 24038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:59:45.338840 24042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:59:45.338860 24031 server_base.cc:1034] running on GCE node
I20260807 02:59:45.339566 24031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:59:45.340569 24031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:59:45.341826 24031 hybrid_clock.cc:648] HybridClock initialized: now 1786071585341734 us; error 85 us; skew 500 ppm
I20260807 02:59:45.342173 24031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 02:59:45 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071585, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:59:45.345649 24031 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 02:59:45.347142 24031 webserver.cc:460] Webserver started at http://127.1.121.67:33553/ using document root <none> and password file <none>
I20260807 02:59:45.347832 24031 fs_manager.cc:359] Metadata directory not provided
I20260807 02:59:45.347936 24031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:59:45.348209 24031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:59:45.351308 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "ceb7c4f6b8ce431d91be85ce872ada63"
format_stamp: "Formatted at 2026-08-07 02:59:45 on dist-test-slave-3h5h"
server_key: "ab243adbe9c12ad6c9fca3d39a6a03bb"
server_key_iv: "8c16a6a7cd43c0bfa4b0b7059a1355b3"
server_key_version: "encryptionkey@0"
I20260807 02:59:45.352180 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "ceb7c4f6b8ce431d91be85ce872ada63"
format_stamp: "Formatted at 2026-08-07 02:59:45 on dist-test-slave-3h5h"
server_key: "ab243adbe9c12ad6c9fca3d39a6a03bb"
server_key_iv: "8c16a6a7cd43c0bfa4b0b7059a1355b3"
server_key_version: "encryptionkey@0"
I20260807 02:59:45.357722 24031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 02:59:45.361579 24051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:45.363073 24031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:59:45.363214 24031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "ceb7c4f6b8ce431d91be85ce872ada63"
format_stamp: "Formatted at 2026-08-07 02:59:45 on dist-test-slave-3h5h"
server_key: "ab243adbe9c12ad6c9fca3d39a6a03bb"
server_key_iv: "8c16a6a7cd43c0bfa4b0b7059a1355b3"
server_key_version: "encryptionkey@0"
I20260807 02:59:45.363389 24031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:59:45.382591 24031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:59:45.387243 24031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:59:45.387501 24031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:59:45.388584 24031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:59:45.390049 24031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:59:45.390128 24031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:45.390218 24031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:59:45.390264 24031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:59:45.427706 24031 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:38517
I20260807 02:59:45.427757 24226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:38517 every 8 connection(s)
I20260807 02:59:45.429205 24031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 02:59:45.433816 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 24031
I20260807 02:59:45.434026 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 02:59:45.434526 1509 external_mini_cluster.cc:1428] Setting key 810e10f1c3eb00fce3d689f9b0402991
Aug 07 02:59:45 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071585, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 02:59:45.451608 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.431929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:40221 (local address 127.1.121.126:37333)
0807 02:59:45.432415 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:45.432420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:45.433479 (+ 1059us) server_negotiation.cc:407] Connection header received
0807 02:59:45.435990 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:45.435997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:45.436099 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:45.436236 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:45.437963 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.438816 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.440543 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.441068 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.441217 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.445068 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:45.445074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:45.445078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:45.445114 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:45.448006 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:45.448819 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:45.448878 (+ 59us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:45.448884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:45.449042 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:45.449593 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:45.449597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:45.449600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:45.449940 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:45.450140 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:45.450909 (+ 769us) server_negotiation.cc:299] Negotiation successful
0807 02:59:45.451171 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260807 02:59:45.452744 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.432483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:40221)
0807 02:59:45.433301 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:45.433342 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:45.435689 (+ 2347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:45.436530 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:45.436539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:45.437161 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:59:45.437772 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.437790 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.439015 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.439019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.440365 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.440379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.441333 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.441342 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.441600 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.442364 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:59:45.442389 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:59:45.444768 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:59:45.448170 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:45.448180 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:45.448198 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:45.448666 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:45.449172 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:59:45.449178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:59:45.449182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:59:45.449419 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:59:45.450102 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:59:45.450110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:59:45.450495 (+ 385us) client_negotiation.cc:769] Sending connection context
0807 02:59:45.450798 (+ 303us) client_negotiation.cc:240] Negotiation successful
0807 02:59:45.451405 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":356,"threads_started":1}
I20260807 02:59:45.454216 24227 heartbeater.cc:347] Connected to a master server at 127.1.121.126:37333
I20260807 02:59:45.454543 24227 heartbeater.cc:464] Registering TS with master...
I20260807 02:59:45.455313 24227 heartbeater.cc:511] Master 127.1.121.126:37333 requested a full tablet report, sending...
I20260807 02:59:45.456748 23526 ts_manager.cc:194] Registered new tserver with Master: ceb7c4f6b8ce431d91be85ce872ada63 (127.1.121.67:38517)
I20260807 02:59:45.457682 23526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:40221
I20260807 02:59:45.462522 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:59:45.482450 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.467768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60950 (local address 127.1.121.126:37333)
0807 02:59:45.468044 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:45.468055 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:45.468442 (+ 387us) server_negotiation.cc:407] Connection header received
0807 02:59:45.468594 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:45.468599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:45.468670 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:45.468760 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:45.470242 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.471145 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.473095 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.473965 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.474162 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.476785 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:45.476790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:45.476794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:45.476827 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:45.479548 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:45.480330 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:45.480342 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:45.480344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:45.480483 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:45.480905 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:45.480911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:45.480915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:45.481283 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:45.481504 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:45.481879 (+ 375us) server_negotiation.cc:299] Negotiation successful
0807 02:59:45.482122 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:59:45.487107 23526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:59:45.494138 23526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:59:45.527544 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.507465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:38517 (local address 127.0.0.1:33484)
0807 02:59:45.518739 (+ 11274us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:45.518805 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:45.519027 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:45.520243 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:45.520253 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:45.520292 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:45.520795 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.520809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.522628 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.522638 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.524935 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.524953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.526330 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.526336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.526632 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.526726 (+ 94us) client_negotiation.cc:769] Sending connection context
0807 02:59:45.526848 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 02:59:45.527070 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10932,"mutex_wait_us":4,"thread_start_us":349,"threads_started":1}
I20260807 02:59:45.528084 24247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.507844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.1.121.67:38517)
0807 02:59:45.518960 (+ 11116us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:45.518971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:45.519011 (+ 40us) server_negotiation.cc:407] Connection header received
0807 02:59:45.519332 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:45.519345 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:45.519644 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:45.519888 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:45.521044 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.522469 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.525103 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.526090 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.526265 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.526535 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:45.527381 (+ 846us) server_negotiation.cc:299] Negotiation successful
0807 02:59:45.527572 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10899,"thread_start_us":11186,"threads_started":1}
I20260807 02:59:45.530164 24248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.508011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:39309 (local address 127.0.0.1:33160)
0807 02:59:45.520220 (+ 12209us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:45.520240 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:45.520363 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:45.521269 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:45.521275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:45.521303 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:45.521723 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.521737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.523750 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.523755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.526494 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.526510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.529308 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.529315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.529487 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.529532 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 02:59:45.529647 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 02:59:45.529808 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12044,"thread_start_us":11877,"threads_started":1}
I20260807 02:59:45.530694 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.509785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.1.121.65:39757)
0807 02:59:45.521251 (+ 11466us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:45.521260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:45.521288 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:59:45.521391 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:45.521401 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:45.521664 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:45.521899 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:45.522509 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.524154 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.528110 (+ 3956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.529490 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.529653 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.529892 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:45.530095 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 02:59:45.530290 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11274,"thread_start_us":11166,"threads_started":1}
I20260807 02:59:45.531338 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.508323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33160 (local address 127.1.121.66:39309)
0807 02:59:45.519697 (+ 11374us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:45.519708 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:45.520524 (+ 816us) server_negotiation.cc:407] Connection header received
0807 02:59:45.520630 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:45.520637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:45.520909 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:45.521100 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:45.521939 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.523578 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.526649 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.528142 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.529261 (+ 1119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.529470 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:45.530486 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0807 02:59:45.530697 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11203,"thread_start_us":11107,"threads_started":1}
I20260807 02:59:45.530547 24245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.509324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39757 (local address 127.0.0.1:52534)
0807 02:59:45.520659 (+ 11335us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:45.520679 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:45.520850 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:45.521909 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:45.521914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:45.521941 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:45.522288 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.522298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.524323 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.524330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.527093 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.527107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.529726 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.529732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.529884 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.529930 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 02:59:45.530023 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 02:59:45.530142 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11187,"mutex_wait_us":1,"thread_start_us":8884,"threads_started":1}
I20260807 02:59:45.532799 24133 tablet_service.cc:1467] Processing CreateTablet for tablet 68081c087eae4dd3b8f6f7fa8a5f0b59 (DEFAULT_TABLE table=test-table [id=6f14ddca063a4b58a268490cc965402e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:45.534669 24133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68081c087eae4dd3b8f6f7fa8a5f0b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:45.534929 23921 tablet_service.cc:1467] Processing CreateTablet for tablet 68081c087eae4dd3b8f6f7fa8a5f0b59 (DEFAULT_TABLE table=test-table [id=6f14ddca063a4b58a268490cc965402e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:45.535413 23712 tablet_service.cc:1467] Processing CreateTablet for tablet 68081c087eae4dd3b8f6f7fa8a5f0b59 (DEFAULT_TABLE table=test-table [id=6f14ddca063a4b58a268490cc965402e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:59:45.536321 23921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68081c087eae4dd3b8f6f7fa8a5f0b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:45.537391 23712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68081c087eae4dd3b8f6f7fa8a5f0b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:59:45.547334 24253 tablet_bootstrap.cc:492] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: Bootstrap starting.
I20260807 02:59:45.551205 24253 tablet_bootstrap.cc:654] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:45.551545 24254 tablet_bootstrap.cc:492] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: Bootstrap starting.
I20260807 02:59:45.552876 24253 log.cc:824] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:45.554144 24255 tablet_bootstrap.cc:492] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: Bootstrap starting.
I20260807 02:59:45.556393 24254 tablet_bootstrap.cc:654] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:45.557499 24255 tablet_bootstrap.cc:654] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: Neither blocks nor log segments found. Creating new log.
I20260807 02:59:45.558190 24254 log.cc:824] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:45.558840 24255 log.cc:824] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: Log is configured to *not* fsync() on all Append() calls
I20260807 02:59:45.560803 24253 tablet_bootstrap.cc:492] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: No bootstrap required, opened a new log
I20260807 02:59:45.561304 24253 ts_tablet_manager.cc:1397] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260807 02:59:45.565387 24254 tablet_bootstrap.cc:492] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: No bootstrap required, opened a new log
I20260807 02:59:45.565476 24255 tablet_bootstrap.cc:492] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: No bootstrap required, opened a new log
I20260807 02:59:45.565743 24254 ts_tablet_manager.cc:1397] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260807 02:59:45.565814 24255 ts_tablet_manager.cc:1397] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260807 02:59:45.574373 24253 raft_consensus.cc:348] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } }
I20260807 02:59:45.574946 24253 raft_consensus.cc:374] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:45.575078 24253 raft_consensus.cc:729] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd3f1f186fb4d62b4374ab35e776d4b, State: Initialized, Role: FOLLOWER
I20260807 02:59:45.574772 24254 raft_consensus.cc:348] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } }
I20260807 02:59:45.575263 24254 raft_consensus.cc:374] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:45.575402 24254 raft_consensus.cc:729] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb7c4f6b8ce431d91be85ce872ada63, State: Initialized, Role: FOLLOWER
I20260807 02:59:45.576042 24254 consensus_queue.cc:260] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } }
I20260807 02:59:45.575954 24253 consensus_queue.cc:260] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } }
I20260807 02:59:45.577265 24227 heartbeater.cc:503] Master 127.1.121.126:37333 was elected leader, sending a full tablet report...
I20260807 02:59:45.578008 24254 ts_tablet_manager.cc:1428] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260807 02:59:45.579102 24255 raft_consensus.cc:348] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } }
I20260807 02:59:45.579747 24255 raft_consensus.cc:374] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:59:45.579895 24255 raft_consensus.cc:729] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5221b080598044aea73202497087d5cf, State: Initialized, Role: FOLLOWER
I20260807 02:59:45.580770 24255 consensus_queue.cc:260] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } }
I20260807 02:59:45.583961 24255 ts_tablet_manager.cc:1428] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260807 02:59:45.584350 24020 heartbeater.cc:503] Master 127.1.121.126:37333 was elected leader, sending a full tablet report...
I20260807 02:59:45.585511 24253 ts_tablet_manager.cc:1428] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: Time spent starting tablet: real 0.024s user 0.012s sys 0.010s
W20260807 02:59:45.606424 24021 tablet.cc:2369] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:59:45.681073 24228 tablet.cc:2369] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:45.751600 24264 raft_consensus.cc:482] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:59:45.752010 24264 raft_consensus.cc:504] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } }
I20260807 02:59:45.754416 24264 leader_election.cc:288] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceb7c4f6b8ce431d91be85ce872ada63 (127.1.121.67:38517), abd3f1f186fb4d62b4374ab35e776d4b (127.1.121.66:39309)
I20260807 02:59:45.763782 24247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.754734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:59671 (local address 127.1.121.67:38517)
0807 02:59:45.755009 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:45.755019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:45.755544 (+ 525us) server_negotiation.cc:407] Connection header received
0807 02:59:45.755888 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:45.755895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:45.755975 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:45.756071 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:45.756828 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.758042 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.761225 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.762525 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.762696 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.762773 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:45.763386 (+ 613us) server_negotiation.cc:299] Negotiation successful
0807 02:59:45.763517 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 02:59:45.763787 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.755553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:42149 (local address 127.1.121.66:39309)
0807 02:59:45.755851 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:45.755857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:45.755879 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:59:45.756073 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:45.756078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:45.756152 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:45.756256 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:59:45.756943 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.758042 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.761225 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.762537 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.762705 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.762773 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:45.763386 (+ 613us) server_negotiation.cc:299] Negotiation successful
0807 02:59:45.763517 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260807 02:59:45.764076 24266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.754639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:38517 (local address 127.1.121.65:59671)
0807 02:59:45.755424 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:45.755445 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:45.755688 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:45.756105 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:45.756109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:45.756135 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:45.756668 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.756684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.758243 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.758248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.761053 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.761061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.762797 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.762806 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.763128 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.763231 (+ 103us) client_negotiation.cc:769] Sending connection context
0807 02:59:45.763331 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 02:59:45.763582 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":274,"spinlock_wait_cycles":25856,"thread_start_us":364,"threads_started":1}
I20260807 02:59:45.764146 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.754940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:39309 (local address 127.1.121.65:42149)
0807 02:59:45.755802 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0807 02:59:45.755821 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:59:45.755937 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:59:45.756289 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:59:45.756293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:59:45.756326 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:59:45.756695 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.756704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.758246 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.758250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.761045 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:59:45.761061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.762814 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.762819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:59:45.763125 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.763231 (+ 106us) client_negotiation.cc:769] Sending connection context
0807 02:59:45.763331 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 02:59:45.763582 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":461,"thread_start_us":494,"threads_started":1}
I20260807 02:59:45.765256 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68081c087eae4dd3b8f6f7fa8a5f0b59" candidate_uuid: "5221b080598044aea73202497087d5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" is_pre_election: true
I20260807 02:59:45.765256 23956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68081c087eae4dd3b8f6f7fa8a5f0b59" candidate_uuid: "5221b080598044aea73202497087d5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" is_pre_election: true
I20260807 02:59:45.765863 23956 raft_consensus.cc:2454] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5221b080598044aea73202497087d5cf in term 0.
I20260807 02:59:45.765863 24165 raft_consensus.cc:2454] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5221b080598044aea73202497087d5cf in term 0.
I20260807 02:59:45.766597 23650 leader_election.cc:302] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5221b080598044aea73202497087d5cf, ceb7c4f6b8ce431d91be85ce872ada63; no voters:
I20260807 02:59:45.767020 24264 raft_consensus.cc:2781] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:59:45.767210 24264 raft_consensus.cc:482] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:59:45.767338 24264 raft_consensus.cc:3037] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:45.769722 24264 raft_consensus.cc:504] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } }
I20260807 02:59:45.770350 24264 leader_election.cc:288] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [CANDIDATE]: Term 1 election: Requested vote from peers ceb7c4f6b8ce431d91be85ce872ada63 (127.1.121.67:38517), abd3f1f186fb4d62b4374ab35e776d4b (127.1.121.66:39309)
I20260807 02:59:45.770845 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68081c087eae4dd3b8f6f7fa8a5f0b59" candidate_uuid: "5221b080598044aea73202497087d5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb7c4f6b8ce431d91be85ce872ada63"
I20260807 02:59:45.770908 23956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68081c087eae4dd3b8f6f7fa8a5f0b59" candidate_uuid: "5221b080598044aea73202497087d5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd3f1f186fb4d62b4374ab35e776d4b"
I20260807 02:59:45.771106 24165 raft_consensus.cc:3037] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:45.771142 23956 raft_consensus.cc:3037] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:59:45.773656 24165 raft_consensus.cc:2454] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5221b080598044aea73202497087d5cf in term 1.
I20260807 02:59:45.773663 23956 raft_consensus.cc:2454] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5221b080598044aea73202497087d5cf in term 1.
I20260807 02:59:45.774155 23649 leader_election.cc:302] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5221b080598044aea73202497087d5cf, abd3f1f186fb4d62b4374ab35e776d4b; no voters:
I20260807 02:59:45.774511 24264 raft_consensus.cc:2781] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:59:45.775058 24264 raft_consensus.cc:686] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 1 LEADER]: Becoming Leader. State: Replica: 5221b080598044aea73202497087d5cf, State: Running, Role: LEADER
I20260807 02:59:45.775626 24264 consensus_queue.cc:237] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } }
I20260807 02:59:45.780395 23526 catalog_manager.cc:5495] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5221b080598044aea73202497087d5cf (127.1.121.65). New cstate: current_term: 1 leader_uuid: "5221b080598044aea73202497087d5cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } health_report { overall_health: HEALTHY } } }
W20260807 02:59:45.793591 23816 tablet.cc:2369] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:59:45.827901 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.822686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.1.121.65:39757)
0807 02:59:45.822888 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:45.822893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:45.822912 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:59:45.823117 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:45.823120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:45.823205 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:45.823346 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:59:45.823728 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.824428 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.825898 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.826301 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.826396 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.826893 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:59:45.827414 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:59:45.827522 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:45.827626 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 02:59:45.827701 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 02:59:45.836447 23956 raft_consensus.cc:1261] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [term 1 FOLLOWER]: Refusing update from remote peer 5221b080598044aea73202497087d5cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:45.836445 24165 raft_consensus.cc:1261] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [term 1 FOLLOWER]: Refusing update from remote peer 5221b080598044aea73202497087d5cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:59:45.837566 24268 consensus_queue.cc:1035] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:45.837949 24264 consensus_queue.cc:1035] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:59:45.851317 24272 mvcc.cc:205] Tried to move back new op lower bound from 7315749215575490560 to 7315749215341473792. Current Snapshot: MvccSnapshot[applied={T|T < 7315749215575490560}]
I20260807 02:59:45.860139 24276 mvcc.cc:205] Tried to move back new op lower bound from 7315749215575490560 to 7315749215341473792. Current Snapshot: MvccSnapshot[applied={T|T < 7315749215575490560}]
I20260807 02:59:45.871837 23525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:59:45.872267 23525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:59:45.876796 23525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Sending DeleteTablet for 3 replicas of tablet 68081c087eae4dd3b8f6f7fa8a5f0b59
I20260807 02:59:45.878146 23712 tablet_service.cc:1514] Processing DeleteTablet for tablet 68081c087eae4dd3b8f6f7fa8a5f0b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:45 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:52534
I20260807 02:59:45.878119 24133 tablet_service.cc:1514] Processing DeleteTablet for tablet 68081c087eae4dd3b8f6f7fa8a5f0b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:45 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:33484
I20260807 02:59:45.878181 23921 tablet_service.cc:1514] Processing DeleteTablet for tablet 68081c087eae4dd3b8f6f7fa8a5f0b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:59:45 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:33160
I20260807 02:59:45.882690 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 23620
I20260807 02:59:45.891875 24292 tablet_replica.cc:330] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: stopping tablet replica
I20260807 02:59:45.893126 24290 tablet_replica.cc:330] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: stopping tablet replica
I20260807 02:59:45.893527 24290 raft_consensus.cc:2229] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:59:45.894034 24290 raft_consensus.cc:2258] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:45.892510 24292 raft_consensus.cc:2229] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:59:45.896720 24292 raft_consensus.cc:2258] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:45.903405 24290 ts_tablet_manager.cc:1905] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:45.903455 24292 ts_tablet_manager.cc:1905] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:45.904177 24293 tablet_replica.cc:330] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: stopping tablet replica
I20260807 02:59:45.904657 24293 raft_consensus.cc:2229] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:59:45.905236 24293 raft_consensus.cc:2258] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:59:45.907624 24293 ts_tablet_manager.cc:1905] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:59:45.911849 24292 ts_tablet_manager.cc:1918] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:45.912081 24292 log.cc:1196] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/68081c087eae4dd3b8f6f7fa8a5f0b59
I20260807 02:59:45.912587 24290 ts_tablet_manager.cc:1918] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:45.912757 24290 log.cc:1196] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/68081c087eae4dd3b8f6f7fa8a5f0b59
I20260807 02:59:45.912781 24292 ts_tablet_manager.cc:1939] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P 5221b080598044aea73202497087d5cf: Deleting consensus metadata
I20260807 02:59:45.913286 24290 ts_tablet_manager.cc:1939] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P ceb7c4f6b8ce431d91be85ce872ada63: Deleting consensus metadata
I20260807 02:59:45.915104 23503 catalog_manager.cc:4841] TS 5221b080598044aea73202497087d5cf (127.1.121.65:39757): tablet 68081c087eae4dd3b8f6f7fa8a5f0b59 (table test-table [id=6f14ddca063a4b58a268490cc965402e]) successfully deleted
I20260807 02:59:45.915153 23505 catalog_manager.cc:4841] TS ceb7c4f6b8ce431d91be85ce872ada63 (127.1.121.67:38517): tablet 68081c087eae4dd3b8f6f7fa8a5f0b59 (table test-table [id=6f14ddca063a4b58a268490cc965402e]) successfully deleted
Aug 07 02:59:45 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 02:59:45.919816 24293 ts_tablet_manager.cc:1918] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:59:45.920024 24293 log.cc:1196] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/68081c087eae4dd3b8f6f7fa8a5f0b59
I20260807 02:59:45.920583 24293 ts_tablet_manager.cc:1939] T 68081c087eae4dd3b8f6f7fa8a5f0b59 P abd3f1f186fb4d62b4374ab35e776d4b: Deleting consensus metadata
I20260807 02:59:45.922673 23504 catalog_manager.cc:4841] TS abd3f1f186fb4d62b4374ab35e776d4b (127.1.121.66:39309): tablet 68081c087eae4dd3b8f6f7fa8a5f0b59 (table test-table [id=6f14ddca063a4b58a268490cc965402e]) successfully deleted
I20260807 02:59:45.929106 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:45.884209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.1.121.65:39757)
0807 02:59:45.884408 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:45.884416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:45.889619 (+ 5203us) server_negotiation.cc:407] Connection header received
0807 02:59:45.889877 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:45.889883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:45.890011 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:45.890126 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:45.895718 (+ 5592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.897118 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.900382 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:45.901185 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:45.901348 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:45.921371 (+ 20023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:45.921386 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:45.921403 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:45.921445 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:45.925599 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:45.926405 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:45.926416 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:45.926420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:45.926616 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:45.927179 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:45.927185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:45.927188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:45.927691 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:45.928005 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:45.928401 (+ 396us) server_negotiation.cc:299] Negotiation successful
0807 02:59:45.928670 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 02:59:45.929925 23810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:46.036046 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 23620
I20260807 02:59:46.066851 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 23823
Aug 07 02:59:46 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 02:59:46.083947 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:46.067864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.1.121.66:39309)
0807 02:59:46.068039 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:46.068044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:46.068064 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:59:46.068220 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:46.068224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:46.068288 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:46.068374 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:46.070044 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:46.070824 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:46.072549 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:46.073105 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:46.073228 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:46.077829 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:46.077843 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:46.077861 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:46.077897 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:46.080895 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:46.081801 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:46.081808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:46.081811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:46.081939 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:46.082312 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:46.082315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:46.082317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:46.082702 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:46.083054 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:46.083386 (+ 332us) server_negotiation.cc:299] Negotiation successful
0807 02:59:46.083557 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:59:46.084700 24015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:46.175905 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 23823
I20260807 02:59:46.199085 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 24031
Aug 07 02:59:46 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 02:59:46.213999 24247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:59:46.199974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.1.121.67:38517)
0807 02:59:46.200163 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 02:59:46.200168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:59:46.200189 (+ 21us) server_negotiation.cc:407] Connection header received
0807 02:59:46.200346 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:59:46.200351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:59:46.200414 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:59:46.200502 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:59:46.201814 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:46.202507 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:46.203925 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:59:46.204299 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:59:46.204391 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:59:46.208724 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:59:46.208739 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:59:46.208758 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:59:46.208800 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:59:46.211416 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:46.212047 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:46.212054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:46.212056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:46.212197 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:59:46.212521 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:59:46.212524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:59:46.212526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:59:46.212964 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:59:46.213193 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 02:59:46.213437 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 02:59:46.213596 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:59:46.214829 24219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:46.306247 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 24031
I20260807 02:59:46.330996 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 23478
I20260807 02:59:46.332190 23569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 02:59:46.443985 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 23478
I20260807 03:00:06.474685 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37333
--webserver_interface=127.1.121.126
--webserver_port=34567
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:37333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:06.831441 24304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:06.831828 24304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:06.831998 24304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:06.839612 24304 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 03:00:06.839779 24304 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 03:00:06.839879 24304 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 03:00:06.839972 24304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:00:06.840075 24304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:06.840201 24304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:06.840298 24304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:00:06.840380 24304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:00:06.850613 24304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:37333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=34567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:06.852041 24304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:06.853492 24304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:06.881675 24313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:06.881743 24317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:06.881765 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:06.882431 24304 server_base.cc:1034] running on GCE node
I20260807 03:00:06.883436 24304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:06.885223 24304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:06.886492 24304 hybrid_clock.cc:648] HybridClock initialized: now 1786071606886399 us; error 64 us; skew 500 ppm
I20260807 03:00:06.886916 24304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:06 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:06.896127 24304 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:00:06.898043 24304 webserver.cc:460] Webserver started at http://127.1.121.126:34567/ using document root <none> and password file <none>
I20260807 03:00:06.898898 24304 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:06.899065 24304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:06.905747 24304 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:00:06.908973 24325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:06.910480 24304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:00:06.910686 24304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "cde83b0ae3e74ce8ae4cbb38103455ec"
format_stamp: "Formatted at 2026-08-07 02:59:43 on dist-test-slave-3h5h"
server_key: "95e59efdf7d598d2defd977309d47a9c"
server_key_iv: "7c94d60409cb44a7500c28a85845d7eb"
server_key_version: "encryptionkey@0"
I20260807 03:00:06.911783 24304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:06.927943 24304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:06.932336 24304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:06.932660 24304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:06.954618 24304 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:37333
I20260807 03:00:06.954671 24400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:37333 every 8 connection(s)
I20260807 03:00:06.956888 24304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:00:06.957814 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 24304
I20260807 03:00:06.958714 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:39757
--local_ip_for_outbound_sockets=127.1.121.65
--tserver_master_addrs=127.1.121.126:37333
--webserver_port=42421
--webserver_interface=127.1.121.65
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:00:06.966260 24402 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:00:06.972162 24402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Bootstrap starting.
I20260807 03:00:06.995517 24402 log.cc:824] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:07.008527 24402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:00:07.009199 24402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Bootstrap complete.
I20260807 03:00:07.019248 24402 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 03:00:07.020181 24402 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde83b0ae3e74ce8ae4cbb38103455ec, State: Initialized, Role: FOLLOWER
I20260807 03:00:07.020967 24402 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 03:00:07.021236 24402 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:00:07.021373 24402 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:00:07.021577 24402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:00:07.025702 24402 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 03:00:07.026273 24402 leader_election.cc:302] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde83b0ae3e74ce8ae4cbb38103455ec; no voters:
I20260807 03:00:07.026970 24402 leader_election.cc:288] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 03:00:07.027150 24408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:00:07.028402 24408 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 2 LEADER]: Becoming Leader. State: Replica: cde83b0ae3e74ce8ae4cbb38103455ec, State: Running, Role: LEADER
I20260807 03:00:07.029026 24408 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 03:00:07.029429 24402 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:00:07.031919 24412 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde83b0ae3e74ce8ae4cbb38103455ec. Latest consensus state: current_term: 2 leader_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } } }
I20260807 03:00:07.032418 24412 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: This master's current role is: LEADER
I20260807 03:00:07.033053 24410 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } } }
I20260807 03:00:07.033419 24410 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: This master's current role is: LEADER
I20260807 03:00:07.037621 24417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:00:07.047734 24417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:00:07.049659 24417 catalog_manager.cc:1259] Loaded cluster ID: 69c6c09a3eb644d9a72f86f7a2cee67c
I20260807 03:00:07.049784 24417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:00:07.055104 24417 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:00:07.057600 24417 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Loaded TSK: 0
I20260807 03:00:07.103899 24417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Generated new TSK 1
I20260807 03:00:07.104410 24417 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 03:00:07.389657 24405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:07.390033 24405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:07.390198 24405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:07.397758 24405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:07.397907 24405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:07.398087 24405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 03:00:07.408068 24405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:39757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=42421
--tserver_master_addrs=127.1.121.126:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:07.409631 24405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:07.411023 24405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:07.426795 24446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:07.426762 24447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:07.427762 24449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:07.427903 24405 server_base.cc:1034] running on GCE node
I20260807 03:00:07.428567 24405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:07.429863 24405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:07.431085 24405 hybrid_clock.cc:648] HybridClock initialized: now 1786071607431036 us; error 52 us; skew 500 ppm
I20260807 03:00:07.431427 24405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:07 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:07.434779 24405 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 03:00:07.436515 24405 webserver.cc:460] Webserver started at http://127.1.121.65:42421/ using document root <none> and password file <none>
I20260807 03:00:07.437366 24405 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:07.437523 24405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:07.443508 24405 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260807 03:00:07.446501 24459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:07.447980 24405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:00:07.448174 24405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "5221b080598044aea73202497087d5cf"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "2321c16d41a71e9a4d0c228604e71216"
server_key_iv: "a5f423b03e70a38988d4c0b5a59a8ba2"
server_key_version: "encryptionkey@0"
I20260807 03:00:07.449230 24405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:07.480250 24405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:07.485080 24405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:07.485383 24405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:07.486543 24405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:07.487943 24405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:07.488045 24405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:07.488152 24405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:07.488252 24405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:07.522238 24405 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:39757
I20260807 03:00:07.522289 24612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:39757 every 8 connection(s)
I20260807 03:00:07.523813 24405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 03:00:07.534082 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 24405
I20260807 03:00:07.534765 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:39309
--local_ip_for_outbound_sockets=127.1.121.66
--tserver_master_addrs=127.1.121.126:37333
--webserver_port=45163
--webserver_interface=127.1.121.66
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:00:07 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:07.549175 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:07.526856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:38827 (local address 127.1.121.126:37333)
0807 03:00:07.527524 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:07.527544 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:07.528044 (+ 500us) server_negotiation.cc:407] Connection header received
0807 03:00:07.530441 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:07.530501 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:07.530899 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:07.531286 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:07.532884 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:07.534323 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:07.535978 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:07.536708 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:07.537121 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:07.541496 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:07.541505 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:07.541519 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:07.541560 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:07.544597 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:07.545458 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:07.545465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:07.545470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:07.545642 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:07.546099 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:07.546106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:07.546115 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:07.546594 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:07.546801 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:07.547789 (+ 988us) server_negotiation.cc:299] Negotiation successful
0807 03:00:07.548122 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":272,"threads_started":1}
I20260807 03:00:07.549700 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:07.526960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:38827)
0807 03:00:07.527853 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:07.527895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:07.529866 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:07.531428 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:07.531442 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:07.532056 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:07.532599 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:07.532617 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:07.534507 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:07.534512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:07.535813 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:07.535827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:07.537134 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:07.537145 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:07.537486 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:07.538242 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:00:07.538277 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:07.541208 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:07.544795 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:07.544804 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:07.544819 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:07.545298 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:07.545765 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:07.545771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:07.545775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:07.545971 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:07.546741 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:07.546748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:07.547173 (+ 425us) client_negotiation.cc:769] Sending connection context
0807 03:00:07.547498 (+ 325us) client_negotiation.cc:240] Negotiation successful
0807 03:00:07.548125 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":416,"threads_started":1}
I20260807 03:00:07.553268 24613 heartbeater.cc:347] Connected to a master server at 127.1.121.126:37333
I20260807 03:00:07.553644 24613 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:07.554564 24613 heartbeater.cc:511] Master 127.1.121.126:37333 requested a full tablet report, sending...
I20260807 03:00:07.556732 24353 ts_manager.cc:194] Registered new tserver with Master: 5221b080598044aea73202497087d5cf (127.1.121.65:39757)
I20260807 03:00:07.558254 24353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:38827
W20260807 03:00:07.893159 24618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:07.893512 24618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:07.893713 24618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:07.901268 24618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:07.901438 24618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:07.901628 24618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 03:00:07.911721 24618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:39309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=45163
--tserver_master_addrs=127.1.121.126:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:07.913084 24618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:07.914459 24618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:07.928495 24625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:07.928824 24626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:07.928877 24629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:07.930150 24618 server_base.cc:1034] running on GCE node
I20260807 03:00:07.930902 24618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:07.931999 24618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:07.933255 24618 hybrid_clock.cc:648] HybridClock initialized: now 1786071607933200 us; error 58 us; skew 500 ppm
I20260807 03:00:07.933634 24618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:07 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:07.937310 24618 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 03:00:07.939245 24618 webserver.cc:460] Webserver started at http://127.1.121.66:45163/ using document root <none> and password file <none>
I20260807 03:00:07.940385 24618 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:07.940560 24618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:07.946213 24618 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260807 03:00:07.949083 24639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:07.950543 24618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:00:07.950721 24618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "abd3f1f186fb4d62b4374ab35e776d4b"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "bc35f11efd75c19ed823253c9a6267f9"
server_key_iv: "f6a20625006c1316b056bb51c3bd302d"
server_key_version: "encryptionkey@0"
I20260807 03:00:07.951719 24618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:07.970844 24618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:07.975497 24618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:07.975802 24618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:07.976930 24618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:07.978358 24618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:07.978482 24618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:07.978639 24618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:07.978734 24618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:08.013521 24618 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:39309
I20260807 03:00:08.013571 24798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:39309 every 8 connection(s)
I20260807 03:00:08.015117 24618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 03:00:08.020339 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 24618
I20260807 03:00:08.021077 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:38517
--local_ip_for_outbound_sockets=127.1.121.67
--tserver_master_addrs=127.1.121.126:37333
--webserver_port=33553
--webserver_interface=127.1.121.67
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:00:08 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:08.035944 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.017949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:46321 (local address 127.1.121.126:37333)
0807 03:00:08.018158 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.018163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.019298 (+ 1135us) server_negotiation.cc:407] Connection header received
0807 03:00:08.021363 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.021368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.021458 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.021518 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:08.023254 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.023968 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.025683 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.026123 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.026267 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.029937 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:08.029942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:08.029945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:08.029980 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:08.032783 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.033590 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.033597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.033600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.033743 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.034194 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.034198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.034200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.034479 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:08.034643 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.035346 (+ 703us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.035584 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 03:00:08.037030 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.018416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:46321)
0807 03:00:08.019123 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:08.019168 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:08.021020 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:08.021973 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:08.021981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:08.022496 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:08.023060 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.023088 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.024162 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.024166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.025539 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.025553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.026380 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.026389 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.026624 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.027324 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:00:08.027352 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:08.029658 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:08.032991 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:08.033001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:08.033019 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:08.033453 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:08.033858 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:08.033863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:08.033867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:08.034064 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:08.034624 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:08.034633 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:08.034981 (+ 348us) client_negotiation.cc:769] Sending connection context
0807 03:00:08.035253 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 03:00:08.035821 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":277,"threads_started":1}
I20260807 03:00:08.038722 24799 heartbeater.cc:347] Connected to a master server at 127.1.121.126:37333
I20260807 03:00:08.039073 24799 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:08.040035 24799 heartbeater.cc:511] Master 127.1.121.126:37333 requested a full tablet report, sending...
I20260807 03:00:08.041828 24353 ts_manager.cc:194] Registered new tserver with Master: abd3f1f186fb4d62b4374ab35e776d4b (127.1.121.66:39309)
I20260807 03:00:08.042655 24353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:46321
W20260807 03:00:08.383035 24806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:08.383432 24806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:08.383636 24806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:08.391888 24806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:08.392068 24806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:08.392290 24806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 03:00:08.402468 24806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:38517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=33553
--tserver_master_addrs=127.1.121.126:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:08.403923 24806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:08.405702 24806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:08.420146 24817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:08.420226 24815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:08.420802 24813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:08.421614 24806 server_base.cc:1034] running on GCE node
I20260807 03:00:08.422258 24806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:08.423239 24806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:08.424481 24806 hybrid_clock.cc:648] HybridClock initialized: now 1786071608424420 us; error 56 us; skew 500 ppm
I20260807 03:00:08.424815 24806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:08 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071608, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:08.428277 24806 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 03:00:08.430188 24806 webserver.cc:460] Webserver started at http://127.1.121.67:33553/ using document root <none> and password file <none>
I20260807 03:00:08.431077 24806 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:08.431264 24806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:08.437328 24806 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 03:00:08.440552 24828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:08.442005 24806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:00:08.442191 24806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "ceb7c4f6b8ce431d91be85ce872ada63"
format_stamp: "Formatted at 2026-08-07 02:59:45 on dist-test-slave-3h5h"
server_key: "ab243adbe9c12ad6c9fca3d39a6a03bb"
server_key_iv: "8c16a6a7cd43c0bfa4b0b7059a1355b3"
server_key_version: "encryptionkey@0"
I20260807 03:00:08.443198 24806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:08.471983 24806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:08.476366 24806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:08.476665 24806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:08.477799 24806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:08.479203 24806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:08.479326 24806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:08.479450 24806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:08.479535 24806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:08.515031 24806 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:38517
I20260807 03:00:08.515127 24986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:38517 every 8 connection(s)
I20260807 03:00:08.516624 24806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 03:00:08.517431 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 24806
Aug 07 03:00:08 dist-test-slave-3h5h krb5kdc[23456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071608, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:08.536906 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.518823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.1.121.126:37333)
0807 03:00:08.519069 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.519076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.519315 (+ 239us) server_negotiation.cc:407] Connection header received
0807 03:00:08.519596 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.519603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.519719 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.519841 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:08.521731 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.522928 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.525087 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.525750 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.525926 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.530154 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:08.530162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:08.530169 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:08.530206 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:08.533527 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.534327 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.534335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.534344 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.534485 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.534940 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.534946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.534950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.535394 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:08.535739 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.536077 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.536323 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":88}
I20260807 03:00:08.542284 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.519469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:33047 (local address 127.1.121.126:37333)
0807 03:00:08.521071 (+ 1602us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.521079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.521164 (+ 85us) server_negotiation.cc:407] Connection header received
0807 03:00:08.523565 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.523580 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.523661 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.523788 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:08.525484 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.527296 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.530069 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.530602 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.530792 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.534651 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:08.534658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:08.534663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:08.534707 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:08.538299 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.539702 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.539712 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.539719 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.539940 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.540415 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.540419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.540422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.540740 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:08.540966 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.541704 (+ 738us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.541905 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1477,"thread_start_us":245,"threads_started":1}
I20260807 03:00:08.543118 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.520119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:33047)
0807 03:00:08.520995 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:08.521040 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:08.523316 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:08.524031 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:08.524039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:08.524517 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:08.525287 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.525310 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.527503 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.527512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.529885 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.529904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.530856 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.530865 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.531216 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.531861 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:00:08.531885 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:08.534379 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:08.538753 (+ 4374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:08.538763 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:08.538782 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:08.539494 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:08.540096 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:08.540100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:08.540103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:08.540304 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:08.540994 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:08.541000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:08.541319 (+ 319us) client_negotiation.cc:769] Sending connection context
0807 03:00:08.541552 (+ 233us) client_negotiation.cc:240] Negotiation successful
0807 03:00:08.541966 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":365,"threads_started":1}
I20260807 03:00:08.544552 24987 heartbeater.cc:347] Connected to a master server at 127.1.121.126:37333
I20260807 03:00:08.544982 24987 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:08.545847 24987 heartbeater.cc:511] Master 127.1.121.126:37333 requested a full tablet report, sending...
I20260807 03:00:08.547502 24353 ts_manager.cc:194] Registered new tserver with Master: ceb7c4f6b8ce431d91be85ce872ada63 (127.1.121.67:38517)
I20260807 03:00:08.548396 24353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:33047
I20260807 03:00:08.552493 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:00:08.560817 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.553441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.1.121.126:37333)
0807 03:00:08.553645 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.553649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.554201 (+ 552us) server_negotiation.cc:407] Connection header received
0807 03:00:08.554433 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.554438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.554691 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.554875 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:00:08.556140 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.557080 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.558973 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.559361 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.559456 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.560001 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:00:08.560278 (+ 277us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 03:00:08.560554 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40374: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260807 03:00:08.560971 24992 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40374: token expired
I20260807 03:00:08.561453 24613 heartbeater.cc:503] Master 127.1.121.126:37333 was elected leader, sending a full tablet report...
W20260807 03:00:08.561537 24232 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.121.126:37333)
I20260807 03:00:08.575131 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.562280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.1.121.126:37333)
0807 03:00:08.562450 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.562455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.562473 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:00:08.562519 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.562525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.562605 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.562703 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:08.564320 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.565120 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.566503 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.566926 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.567062 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.569852 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:08.569858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:08.569871 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:08.569903 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:08.572231 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.573085 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.573091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.573094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.573228 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.573604 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.573609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.573613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.573941 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:08.574226 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.574563 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.574789 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:00:08.578681 24353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:00:08.586117 24353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:00:08.611315 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.598116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39757 (local address 127.0.0.1:41180)
0807 03:00:08.599821 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:08.599869 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:08.600044 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:08.601316 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:08.601328 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:08.601372 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:08.601888 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.601904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.604196 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.604205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.607178 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.607194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.610213 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.610222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.610503 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.610561 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 03:00:08.610714 (+ 153us) client_negotiation.cc:240] Negotiation successful
0807 03:00:08.610921 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1472,"thread_start_us":1261,"threads_started":1}
I20260807 03:00:08.612344 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.598428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.1.121.65:39757)
0807 03:00:08.599139 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.599151 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.600256 (+ 1105us) server_negotiation.cc:407] Connection header received
0807 03:00:08.600368 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.600375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.600689 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.600938 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:08.602188 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.603978 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.607378 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.609498 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.609688 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.610006 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.611622 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.611876 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":315,"threads_started":1}
I20260807 03:00:08.618873 24523 tablet_service.cc:1467] Processing CreateTablet for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b (DEFAULT_TABLE table=test-table [id=8842c77e67e44d53a69c6117daa91095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:08.620687 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:08.625157 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.599193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:39309 (local address 127.0.0.1:51862)
0807 03:00:08.612762 (+ 13569us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:08.612783 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:08.614065 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:08.617952 (+ 3887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:08.617961 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:08.617998 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:08.618266 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.618284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.619934 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.619940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.622395 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.622411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.624311 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.624318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.624492 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.624540 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 03:00:08.624652 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 03:00:08.624800 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13454,"thread_start_us":16474,"threads_started":1}
I20260807 03:00:08.625466 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.598750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.1.121.66:39309)
0807 03:00:08.617323 (+ 18573us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.617332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.617363 (+ 31us) server_negotiation.cc:407] Connection header received
0807 03:00:08.617473 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.617480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.617723 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.617933 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:08.618479 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.619767 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.622603 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.624107 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.624268 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.624459 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.624713 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.624913 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18401,"thread_start_us":18300,"threads_started":1}
I20260807 03:00:08.626183 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.599548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:38517 (local address 127.0.0.1:34414)
0807 03:00:08.615982 (+ 16434us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:08.616001 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:08.616107 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:08.619589 (+ 3482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:08.619594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:08.619700 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:08.620090 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.620100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.621628 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.621634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.623343 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.623356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.625155 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.625161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.625553 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.625610 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 03:00:08.625701 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 03:00:08.625820 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16302,"mutex_wait_us":885,"thread_start_us":17839,"threads_started":1}
I20260807 03:00:08.626478 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.600112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.1.121.67:38517)
0807 03:00:08.618978 (+ 18866us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.618989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.619022 (+ 33us) server_negotiation.cc:407] Connection header received
0807 03:00:08.619116 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.619123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.619346 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.619540 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:08.620274 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.621446 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.623508 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.624606 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.624721 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.624979 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.625770 (+ 791us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.625939 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18678,"thread_start_us":18514,"threads_started":1}
I20260807 03:00:08.629390 24710 tablet_service.cc:1467] Processing CreateTablet for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b (DEFAULT_TABLE table=test-table [id=8842c77e67e44d53a69c6117daa91095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:08.629655 24897 tablet_service.cc:1467] Processing CreateTablet for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b (DEFAULT_TABLE table=test-table [id=8842c77e67e44d53a69c6117daa91095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:08.630873 24710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:08.630960 24897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:08.637280 25010 tablet_bootstrap.cc:492] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Bootstrap starting.
I20260807 03:00:08.641551 25010 tablet_bootstrap.cc:654] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:08.642932 25010 log.cc:824] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:08.644491 25013 tablet_bootstrap.cc:492] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: Bootstrap starting.
I20260807 03:00:08.646278 25014 tablet_bootstrap.cc:492] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: Bootstrap starting.
I20260807 03:00:08.646553 25010 tablet_bootstrap.cc:492] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: No bootstrap required, opened a new log
I20260807 03:00:08.646860 25010 ts_tablet_manager.cc:1397] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 03:00:08.648321 25013 tablet_bootstrap.cc:654] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:08.649642 25013 log.cc:824] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:08.650118 25014 tablet_bootstrap.cc:654] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:08.651593 25014 log.cc:824] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:08.655076 25013 tablet_bootstrap.cc:492] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: No bootstrap required, opened a new log
I20260807 03:00:08.655102 25014 tablet_bootstrap.cc:492] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: No bootstrap required, opened a new log
I20260807 03:00:08.655365 25014 ts_tablet_manager.cc:1397] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 03:00:08.655387 25013 ts_tablet_manager.cc:1397] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260807 03:00:08.655939 25010 raft_consensus.cc:348] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } }
I20260807 03:00:08.656383 25010 raft_consensus.cc:374] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:08.656522 25010 raft_consensus.cc:729] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5221b080598044aea73202497087d5cf, State: Initialized, Role: FOLLOWER
I20260807 03:00:08.657231 25010 consensus_queue.cc:260] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } }
I20260807 03:00:08.662004 25010 ts_tablet_manager.cc:1428] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Time spent starting tablet: real 0.015s user 0.008s sys 0.003s
I20260807 03:00:08.666440 25013 raft_consensus.cc:348] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } }
I20260807 03:00:08.666440 25014 raft_consensus.cc:348] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } }
I20260807 03:00:08.666903 25013 raft_consensus.cc:374] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:08.666903 25014 raft_consensus.cc:374] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:08.667012 25014 raft_consensus.cc:729] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd3f1f186fb4d62b4374ab35e776d4b, State: Initialized, Role: FOLLOWER
I20260807 03:00:08.667019 25013 raft_consensus.cc:729] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb7c4f6b8ce431d91be85ce872ada63, State: Initialized, Role: FOLLOWER
I20260807 03:00:08.667646 25014 consensus_queue.cc:260] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } }
I20260807 03:00:08.667646 25013 consensus_queue.cc:260] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } }
I20260807 03:00:08.668749 24987 heartbeater.cc:503] Master 127.1.121.126:37333 was elected leader, sending a full tablet report...
I20260807 03:00:08.669245 24799 heartbeater.cc:503] Master 127.1.121.126:37333 was elected leader, sending a full tablet report...
I20260807 03:00:08.669615 25013 ts_tablet_manager.cc:1428] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 03:00:08.669860 25014 ts_tablet_manager.cc:1428] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260807 03:00:08.690518 25020 raft_consensus.cc:482] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:00:08.690877 25020 raft_consensus.cc:504] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } }
I20260807 03:00:08.692524 25020 leader_election.cc:288] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abd3f1f186fb4d62b4374ab35e776d4b (127.1.121.66:39309), ceb7c4f6b8ce431d91be85ce872ada63 (127.1.121.67:38517)
I20260807 03:00:08.702069 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.693129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:59357 (local address 127.1.121.66:39309)
0807 03:00:08.693339 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.693345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.694156 (+ 811us) server_negotiation.cc:407] Connection header received
0807 03:00:08.694365 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.694370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.694437 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.694536 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:08.695378 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.696495 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.699558 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.700867 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.701026 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.701103 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.701665 (+ 562us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.701799 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:00:08.702069 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.693149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:40861 (local address 127.1.121.67:38517)
0807 03:00:08.693339 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.693346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.694155 (+ 809us) server_negotiation.cc:407] Connection header received
0807 03:00:08.694346 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.694351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.694431 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.694536 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:08.695378 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.696494 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.699563 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.700874 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.701030 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.701103 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.701673 (+ 570us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.701802 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:00:08.702394 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.692949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:39309 (local address 127.1.121.65:59357)
0807 03:00:08.693999 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:08.694025 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:08.694238 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:08.694575 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:08.694578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:08.694618 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:08.695215 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.695229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.696639 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.696643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.699379 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.699397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.701120 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.701129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.701386 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.701513 (+ 127us) client_negotiation.cc:769] Sending connection context
0807 03:00:08.701619 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 03:00:08.701806 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":280,"spinlock_wait_cycles":40576,"thread_start_us":662,"threads_started":1}
I20260807 03:00:08.702618 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.692983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:38517 (local address 127.1.121.65:40861)
0807 03:00:08.694035 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:08.694051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:08.694186 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:08.694568 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:08.694572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:08.694616 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:08.695224 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.695233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.696637 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.696643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.699386 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:08.699397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.701152 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.701158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:08.701388 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.701514 (+ 126us) client_negotiation.cc:769] Sending connection context
0807 03:00:08.701618 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 03:00:08.701807 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":270,"spinlock_wait_cycles":14464,"thread_start_us":571,"threads_started":1}
I20260807 03:00:08.703461 24742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce8fde356a4ee2aa8a0a91ccd4493b" candidate_uuid: "5221b080598044aea73202497087d5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" is_pre_election: true
I20260807 03:00:08.703517 24923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce8fde356a4ee2aa8a0a91ccd4493b" candidate_uuid: "5221b080598044aea73202497087d5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" is_pre_election: true
I20260807 03:00:08.703969 24923 raft_consensus.cc:2454] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5221b080598044aea73202497087d5cf in term 0.
I20260807 03:00:08.703966 24742 raft_consensus.cc:2454] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5221b080598044aea73202497087d5cf in term 0.
I20260807 03:00:08.704702 24463 leader_election.cc:302] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5221b080598044aea73202497087d5cf, abd3f1f186fb4d62b4374ab35e776d4b, ceb7c4f6b8ce431d91be85ce872ada63; no voters:
I20260807 03:00:08.705206 25020 raft_consensus.cc:2781] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:00:08.705359 25020 raft_consensus.cc:482] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:00:08.705461 25020 raft_consensus.cc:3037] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:08.707683 25020 raft_consensus.cc:504] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } }
I20260807 03:00:08.708256 25020 leader_election.cc:288] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [CANDIDATE]: Term 1 election: Requested vote from peers abd3f1f186fb4d62b4374ab35e776d4b (127.1.121.66:39309), ceb7c4f6b8ce431d91be85ce872ada63 (127.1.121.67:38517)
I20260807 03:00:08.708727 24923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce8fde356a4ee2aa8a0a91ccd4493b" candidate_uuid: "5221b080598044aea73202497087d5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb7c4f6b8ce431d91be85ce872ada63"
I20260807 03:00:08.708736 24742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce8fde356a4ee2aa8a0a91ccd4493b" candidate_uuid: "5221b080598044aea73202497087d5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd3f1f186fb4d62b4374ab35e776d4b"
I20260807 03:00:08.708946 24923 raft_consensus.cc:3037] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:08.708952 24742 raft_consensus.cc:3037] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:08.711120 24923 raft_consensus.cc:2454] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5221b080598044aea73202497087d5cf in term 1.
I20260807 03:00:08.711215 24742 raft_consensus.cc:2454] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5221b080598044aea73202497087d5cf in term 1.
I20260807 03:00:08.711534 24465 leader_election.cc:302] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5221b080598044aea73202497087d5cf, ceb7c4f6b8ce431d91be85ce872ada63; no voters:
I20260807 03:00:08.711843 25020 raft_consensus.cc:2781] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:00:08.712332 25020 raft_consensus.cc:686] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 1 LEADER]: Becoming Leader. State: Replica: 5221b080598044aea73202497087d5cf, State: Running, Role: LEADER
I20260807 03:00:08.712802 25020 consensus_queue.cc:237] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } }
I20260807 03:00:08.717710 24352 catalog_manager.cc:5495] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5221b080598044aea73202497087d5cf (127.1.121.65). New cstate: current_term: 1 leader_uuid: "5221b080598044aea73202497087d5cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5221b080598044aea73202497087d5cf" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 39757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 } health_report { overall_health: UNKNOWN } } }
I20260807 03:00:08.734822 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.728952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.1.121.65:39757)
0807 03:00:08.729186 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.729192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.729212 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:00:08.729448 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.729455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.729568 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.729746 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:00:08.730152 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.731014 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.732535 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.733024 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.733175 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.733658 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:00:08.734216 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:00:08.734386 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.734467 (+ 81us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.734551 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:00:08.743721 24923 raft_consensus.cc:1261] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [term 1 FOLLOWER]: Refusing update from remote peer 5221b080598044aea73202497087d5cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:00:08.743721 24742 raft_consensus.cc:1261] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [term 1 FOLLOWER]: Refusing update from remote peer 5221b080598044aea73202497087d5cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:00:08.744805 25027 consensus_queue.cc:1035] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb7c4f6b8ce431d91be85ce872ada63" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:00:08.745229 25020 consensus_queue.cc:1035] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "abd3f1f186fb4d62b4374ab35e776d4b" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:00:08.774495 24353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:00:08.774860 24353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:00:08.778961 24353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Sending DeleteTablet for 3 replicas of tablet 70ce8fde356a4ee2aa8a0a91ccd4493b
I20260807 03:00:08.780256 24523 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:08 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:41180
I20260807 03:00:08.780643 24897 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:08 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:34414
I20260807 03:00:08.780694 24710 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:08 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:51862
I20260807 03:00:08.781764 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 24405
I20260807 03:00:08.794557 25042 tablet_replica.cc:330] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: stopping tablet replica
I20260807 03:00:08.795445 25042 raft_consensus.cc:2229] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:00:08.796594 25042 raft_consensus.cc:2258] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:08.799232 25042 ts_tablet_manager.cc:1905] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:08.808502 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.783396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.1.121.65:39757)
0807 03:00:08.783588 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.783594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.783613 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:00:08.783671 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.783676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.783760 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.783858 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:08.785886 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.787167 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.789196 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.789937 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.790062 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.798726 (+ 8664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:08.798743 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:08.798762 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:08.798816 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:08.803207 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.804300 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.804310 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.804314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.804510 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.805274 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.805280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.805284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.805840 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:08.806148 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.807852 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.808112 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:00:08.809386 24607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:08.810433 25045 tablet_replica.cc:330] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: stopping tablet replica
I20260807 03:00:08.810657 25043 tablet_replica.cc:330] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: stopping tablet replica
I20260807 03:00:08.810952 25045 raft_consensus.cc:2229] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:00:08.811045 25043 raft_consensus.cc:2229] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:00:08.811558 25045 raft_consensus.cc:2258] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:08.811558 25043 raft_consensus.cc:2258] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:08.814126 25045 ts_tablet_manager.cc:1905] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:08.814551 25043 ts_tablet_manager.cc:1905] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:08.819993 25043 ts_tablet_manager.cc:1918] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:08.820182 25043 log.cc:1196] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/70ce8fde356a4ee2aa8a0a91ccd4493b
I20260807 03:00:08.820689 25043 ts_tablet_manager.cc:1939] T 70ce8fde356a4ee2aa8a0a91ccd4493b P ceb7c4f6b8ce431d91be85ce872ada63: Deleting consensus metadata
I20260807 03:00:08.822317 25045 ts_tablet_manager.cc:1918] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:08.822502 25045 log.cc:1196] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/70ce8fde356a4ee2aa8a0a91ccd4493b
I20260807 03:00:08.822412 24330 catalog_manager.cc:4841] TS ceb7c4f6b8ce431d91be85ce872ada63 (127.1.121.67:38517): tablet 70ce8fde356a4ee2aa8a0a91ccd4493b (table test-table [id=8842c77e67e44d53a69c6117daa91095]) successfully deleted
I20260807 03:00:08.823091 25045 ts_tablet_manager.cc:1939] T 70ce8fde356a4ee2aa8a0a91ccd4493b P abd3f1f186fb4d62b4374ab35e776d4b: Deleting consensus metadata
I20260807 03:00:08.825182 24329 catalog_manager.cc:4841] TS abd3f1f186fb4d62b4374ab35e776d4b (127.1.121.66:39309): tablet 70ce8fde356a4ee2aa8a0a91ccd4493b (table test-table [id=8842c77e67e44d53a69c6117daa91095]) successfully deleted
I20260807 03:00:08.937451 25042 ts_tablet_manager.cc:1918] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:08.937790 25042 log.cc:1196] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/70ce8fde356a4ee2aa8a0a91ccd4493b
I20260807 03:00:08.938671 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 24405
W20260807 03:00:08.965520 24328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.121.65:39757 (error 108)
I20260807 03:00:08.965684 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 24618
I20260807 03:00:08.967619 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.967084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39757 (local address 127.0.0.1:53856)
0807 03:00:08.967285 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:08.967430 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260807 03:00:08.967875 24328 catalog_manager.cc:4584] TS 5221b080598044aea73202497087d5cf (127.1.121.65:39757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70ce8fde356a4ee2aa8a0a91ccd4493b: Network error: Client connection negotiation failed: client connection to 127.1.121.65:39757: connect: Connection refused (error 111)
I20260807 03:00:08.979864 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:08.966461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.1.121.66:39309)
0807 03:00:08.966656 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:08.966663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:08.966988 (+ 325us) server_negotiation.cc:407] Connection header received
0807 03:00:08.967053 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:08.967057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:08.967151 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:08.967261 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:08.969032 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.970124 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.971444 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:08.971830 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:08.971923 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:08.974644 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:08.974656 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:08.974671 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:08.974701 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:08.977417 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.978112 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.978119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.978121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.978241 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:08.978566 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:08.978575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:08.978577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:08.978942 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:08.979165 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:08.979436 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 03:00:08.979612 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:00:08.980523 24794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:09.021880 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:09.021361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39757 (local address 127.0.0.1:53858)
0807 03:00:09.021605 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:09.021705 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 03:00:09.023095 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:09.022695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39757 (local address 127.0.0.1:53870)
0807 03:00:09.022865 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:09.022948 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 03:00:09.072156 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 24618
I20260807 03:00:09.083669 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:09.083015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39757 (local address 127.0.0.1:53886)
0807 03:00:09.083302 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:09.083449 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 03:00:09.084867 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:09.084412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39757 (local address 127.0.0.1:53896)
0807 03:00:09.084593 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:09.084693 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 03:00:09.100298 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 24806
I20260807 03:00:09.114358 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:09.101160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.1.121.67:38517)
0807 03:00:09.101352 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:09.101359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:09.101417 (+ 58us) server_negotiation.cc:407] Connection header received
0807 03:00:09.101635 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:09.101641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:09.101725 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:09.101860 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:09.103197 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:09.103938 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:09.105443 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:09.105856 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:09.105995 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:09.108744 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:09.108759 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:09.108780 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:09.108814 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:09.111429 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:09.112138 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:09.112149 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:09.112157 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:09.112284 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:09.112691 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:09.112697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:09.112701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:09.113135 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:09.113535 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:09.113780 (+ 245us) server_negotiation.cc:299] Negotiation successful
0807 03:00:09.114004 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:00:09.115029 24979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:09.165931 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:09.165369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39757 (local address 127.0.0.1:53906)
0807 03:00:09.165639 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:09.165736 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 03:00:09.167068 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:09.166636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:39757 (local address 127.0.0.1:53908)
0807 03:00:09.166858 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:09.166936 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 03:00:09.207361 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 24806
I20260807 03:00:09.232137 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 24304
I20260807 03:00:09.233421 24395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:09.356535 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 24304
I20260807 03:00:29.388077 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37333
--webserver_interface=127.1.121.126
--webserver_port=34567
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:37333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:29.743434 25052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:29.743729 25052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:29.743902 25052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:29.751179 25052 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 03:00:29.751286 25052 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 03:00:29.751367 25052 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 03:00:29.751451 25052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:00:29.751549 25052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:29.751663 25052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:29.751745 25052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:00:29.751827 25052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:00:29.761950 25052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:37333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:37333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=34567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:29.763341 25052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:29.764781 25052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:29.777957 25059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:29.777904 25063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:29.777982 25061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:29.778992 25052 server_base.cc:1034] running on GCE node
I20260807 03:00:29.779800 25052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:29.781353 25052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:29.782611 25052 hybrid_clock.cc:648] HybridClock initialized: now 1786071629782541 us; error 45 us; skew 500 ppm
I20260807 03:00:29.782970 25052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:29 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:29.786353 25052 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:00:29.787858 25052 webserver.cc:460] Webserver started at http://127.1.121.126:34567/ using document root <none> and password file <none>
I20260807 03:00:29.788524 25052 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:29.788614 25052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:29.794137 25052 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:00:29.797207 25072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:29.798521 25052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:00:29.798677 25052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "cde83b0ae3e74ce8ae4cbb38103455ec"
format_stamp: "Formatted at 2026-08-07 02:59:43 on dist-test-slave-3h5h"
server_key: "95e59efdf7d598d2defd977309d47a9c"
server_key_iv: "7c94d60409cb44a7500c28a85845d7eb"
server_key_version: "encryptionkey@0"
I20260807 03:00:29.799588 25052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:29.815220 25052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:29.819686 25052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:29.819937 25052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:29.839313 25052 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:37333
I20260807 03:00:29.839352 25146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:37333 every 8 connection(s)
I20260807 03:00:29.841058 25052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:00:29.841835 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 25052
I20260807 03:00:29.842865 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:39757
--local_ip_for_outbound_sockets=127.1.121.65
--tserver_master_addrs=127.1.121.126:37333
--webserver_port=42421
--webserver_interface=127.1.121.65
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:00:29.848619 25147 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:00:29.852093 25147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Bootstrap starting.
I20260807 03:00:29.873461 25147 log.cc:824] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:29.890116 25147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:00:29.890704 25147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Bootstrap complete.
I20260807 03:00:29.900424 25147 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 03:00:29.901418 25147 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde83b0ae3e74ce8ae4cbb38103455ec, State: Initialized, Role: FOLLOWER
I20260807 03:00:29.902134 25147 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 03:00:29.902397 25147 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:00:29.902540 25147 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:00:29.902732 25147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:00:29.907056 25147 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 03:00:29.907557 25147 leader_election.cc:302] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde83b0ae3e74ce8ae4cbb38103455ec; no voters:
I20260807 03:00:29.908254 25147 leader_election.cc:288] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 03:00:29.908527 25151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:00:29.910181 25151 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [term 3 LEADER]: Becoming Leader. State: Replica: cde83b0ae3e74ce8ae4cbb38103455ec, State: Running, Role: LEADER
I20260807 03:00:29.910851 25151 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } }
I20260807 03:00:29.911005 25147 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:00:29.914063 25153 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde83b0ae3e74ce8ae4cbb38103455ec. Latest consensus state: current_term: 3 leader_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } } }
I20260807 03:00:29.914577 25153 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: This master's current role is: LEADER
I20260807 03:00:29.915211 25152 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde83b0ae3e74ce8ae4cbb38103455ec" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 37333 } } }
I20260807 03:00:29.915606 25152 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec [sys.catalog]: This master's current role is: LEADER
I20260807 03:00:29.916906 25166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:00:29.927568 25166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:00:29.929400 25166 catalog_manager.cc:1259] Loaded cluster ID: 69c6c09a3eb644d9a72f86f7a2cee67c
I20260807 03:00:29.929590 25166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:00:29.934334 25166 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:00:29.936291 25166 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Loaded TSK: 0
I20260807 03:00:29.936566 25166 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cde83b0ae3e74ce8ae4cbb38103455ec: Loaded TSK: 1
F20260807 03:00:29.938004 25166 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f288c8839ad google::LogMessage::Fail() at ??:0
@ 0x7f288c88688a google::LogMessage::SendToLog() at ??:0
@ 0x7f288c8834de google::LogMessage::Flush() at ??:0
@ 0x7f288c884a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f28905e77d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f28905d673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f28a22b21c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f28a22b1783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f28a233d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f28a233d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f28a23875e9 std::function<>::operator()() at ??:0
@ 0x7f28a22b5cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f28a22b3c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f28a233b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f28a233b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f288f4cf345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f288f4d291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f288f4d2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f288f4adeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f28927876db start_thread at ??:0
@ 0x7f288b71d61f clone at ??:0
W20260807 03:00:30.293102 25149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:30.293393 25149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:30.293498 25149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:30.301136 25149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:30.301254 25149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:30.301369 25149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 03:00:30.311141 25149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:39757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=42421
--tserver_master_addrs=127.1.121.126:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:30.312383 25149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:30.313736 25149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:30.327433 25230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:30.327349 25227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:30.327507 25228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:30.329054 25149 server_base.cc:1034] running on GCE node
I20260807 03:00:30.329638 25149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:30.330593 25149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:30.331856 25149 hybrid_clock.cc:648] HybridClock initialized: now 1786071630331798 us; error 55 us; skew 500 ppm
I20260807 03:00:30.332188 25149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:30 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:30.335402 25149 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 03:00:30.336961 25149 webserver.cc:460] Webserver started at http://127.1.121.65:42421/ using document root <none> and password file <none>
I20260807 03:00:30.337682 25149 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:30.337828 25149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:30.343174 25149 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 03:00:30.346182 25245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:30.347559 25149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:00:30.347780 25149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "5221b080598044aea73202497087d5cf"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "2321c16d41a71e9a4d0c228604e71216"
server_key_iv: "a5f423b03e70a38988d4c0b5a59a8ba2"
server_key_version: "encryptionkey@0"
I20260807 03:00:30.348771 25149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:30.364027 25149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:30.368770 25149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:30.369096 25149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:30.370191 25149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:30.372072 25252 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260807 03:00:30.377578 25252 ts_tablet_manager.cc:1852] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260807 03:00:30.378624 25252 ts_tablet_manager.cc:1905] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:30.385977 25252 ts_tablet_manager.cc:1918] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:30.386231 25252 log.cc:1196] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/70ce8fde356a4ee2aa8a0a91ccd4493b
I20260807 03:00:30.386701 25252 ts_tablet_manager.cc:1939] T 70ce8fde356a4ee2aa8a0a91ccd4493b P 5221b080598044aea73202497087d5cf: Deleting consensus metadata
I20260807 03:00:30.387125 25149 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260807 03:00:30.387272 25149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.000s
I20260807 03:00:30.387439 25149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:30.387530 25149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:30.424152 25149 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:39757
I20260807 03:00:30.424265 25402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:39757 every 8 connection(s)
I20260807 03:00:30.425593 25149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 03:00:30.427227 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 25149
I20260807 03:00:30.428038 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:39309
--local_ip_for_outbound_sockets=127.1.121.66
--tserver_master_addrs=127.1.121.126:37333
--webserver_port=45163
--webserver_interface=127.1.121.66
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:00:30.431342 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:30.428970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:45339)
0807 03:00:30.429905 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:30.430196 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":347,"threads_started":1}
W20260807 03:00:30.432126 25403 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:37333 (0 consecutive failures): Network error: Failed to ping master at 127.1.121.126:37333: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
I20260807 03:00:30.433140 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:30.432679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:59251)
0807 03:00:30.432895 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:30.432990 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 03:00:30.434150 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:30.433714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:44779)
0807 03:00:30.433884 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:30.433983 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 03:00:30.434413 25403 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 03:00:30.784231 25406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:30.784530 25406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:30.784716 25406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:30.792263 25406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:30.792387 25406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:30.792608 25406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 03:00:30.802758 25406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:39309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=45163
--tserver_master_addrs=127.1.121.126:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:30.804165 25406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:30.805637 25406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:30.819350 25419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:30.819506 25416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:30.819605 25414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:30.820616 25406 server_base.cc:1034] running on GCE node
I20260807 03:00:30.821312 25406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:30.822314 25406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:30.823524 25406 hybrid_clock.cc:648] HybridClock initialized: now 1786071630823453 us; error 62 us; skew 500 ppm
I20260807 03:00:30.823879 25406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:30 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071630, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:30.827425 25406 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 03:00:30.828961 25406 webserver.cc:460] Webserver started at http://127.1.121.66:45163/ using document root <none> and password file <none>
I20260807 03:00:30.829659 25406 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:30.829797 25406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:30.835391 25406 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:00:30.838455 25430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:30.839821 25406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:00:30.839991 25406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "abd3f1f186fb4d62b4374ab35e776d4b"
format_stamp: "Formatted at 2026-08-07 02:59:44 on dist-test-slave-3h5h"
server_key: "bc35f11efd75c19ed823253c9a6267f9"
server_key_iv: "f6a20625006c1316b056bb51c3bd302d"
server_key_version: "encryptionkey@0"
I20260807 03:00:30.841063 25406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:30.870628 25406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:30.875530 25406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:30.875849 25406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:30.876991 25406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:30.878436 25406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:30.878526 25406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:30.878691 25406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:30.878786 25406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:30.913815 25406 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:39309
I20260807 03:00:30.913920 25584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:39309 every 8 connection(s)
I20260807 03:00:30.915414 25406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 03:00:30.920421 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:30.918538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:36619)
0807 03:00:30.919298 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:30.919541 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":289,"threads_started":1}
W20260807 03:00:30.921288 25585 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:37333 (0 consecutive failures): Network error: Failed to ping master at 127.1.121.126:37333: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
I20260807 03:00:30.921839 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 25406
I20260807 03:00:30.922487 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:30.922006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:44643)
0807 03:00:30.922225 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:30.922330 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 03:00:30.922735 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:38517
--local_ip_for_outbound_sockets=127.1.121.67
--tserver_master_addrs=127.1.121.126:37333
--webserver_port=33553
--webserver_interface=127.1.121.67
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:00:30.923741 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:30.923260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:38307)
0807 03:00:30.923450 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:30.923559 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 03:00:30.924049 25585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 03:00:31.281041 25589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:31.281358 25589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:31.281563 25589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:31.289207 25589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:31.289374 25589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:31.289585 25589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 03:00:31.303927 25589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:38517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=33553
--tserver_master_addrs=127.1.121.126:37333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:31.305891 25589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:31.308075 25589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:31.328868 25595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:31.329263 25599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:31.329303 25596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:31.331212 25589 server_base.cc:1034] running on GCE node
I20260807 03:00:31.332237 25589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:31.333621 25589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:31.334954 25589 hybrid_clock.cc:648] HybridClock initialized: now 1786071631334879 us; error 72 us; skew 500 ppm
I20260807 03:00:31.335354 25589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:31 dist-test-slave-3h5h krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071631, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:31.338641 25589 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 03:00:31.340097 25589 webserver.cc:460] Webserver started at http://127.1.121.67:33553/ using document root <none> and password file <none>
I20260807 03:00:31.340745 25589 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:31.340914 25589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:31.346376 25589 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 03:00:31.349195 25608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:31.350540 25589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:00:31.350693 25589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "ceb7c4f6b8ce431d91be85ce872ada63"
format_stamp: "Formatted at 2026-08-07 02:59:45 on dist-test-slave-3h5h"
server_key: "ab243adbe9c12ad6c9fca3d39a6a03bb"
server_key_iv: "8c16a6a7cd43c0bfa4b0b7059a1355b3"
server_key_version: "encryptionkey@0"
I20260807 03:00:31.351675 25589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:31.366640 25589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:31.370509 25589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:31.370752 25589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:31.371675 25589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:31.373083 25589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:31.373174 25589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:31.373255 25589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:31.373313 25589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:31.407744 25589 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:38517
I20260807 03:00:31.407835 25765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:38517 every 8 connection(s)
I20260807 03:00:31.409329 25589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 03:00:31.414386 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:31.412476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:41995)
0807 03:00:31.413259 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:31.413479 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":338,"threads_started":1}
W20260807 03:00:31.415099 25766 heartbeater.cc:644] Failed to heartbeat to 127.1.121.126:37333 (0 consecutive failures): Network error: Failed to ping master at 127.1.121.126:37333: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
I20260807 03:00:31.415944 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:31.415628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:52079)
0807 03:00:31.415775 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:31.415844 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 03:00:31.416596 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 25589
I20260807 03:00:31.416963 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:31.416522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:42259)
0807 03:00:31.416697 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:31.416798 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 03:00:31.417294 25766 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 03:00:31.435887 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:31.435041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:60399)
0807 03:00:31.435558 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:31.435657 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":238,"threads_started":1}
I20260807 03:00:31.925519 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:31.924632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:49381)
0807 03:00:31.925205 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:31.925291 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":314,"threads_started":1}
I20260807 03:00:32.418792 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:32.417813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:33639)
0807 03:00:32.418422 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:32.418544 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":380,"threads_started":1}
I20260807 03:00:32.437836 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:32.436972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:46617)
0807 03:00:32.437521 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:32.437615 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":320,"threads_started":1}
I20260807 03:00:32.927273 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:32.926491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:33727)
0807 03:00:32.927004 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:32.927091 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":260,"threads_started":1}
I20260807 03:00:33.420710 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:33.419950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:47789)
0807 03:00:33.420440 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:33.420536 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":242,"threads_started":1}
I20260807 03:00:33.439522 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:33.438784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:34783)
0807 03:00:33.439260 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:33.439348 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
I20260807 03:00:33.929006 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:33.928131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:33125)
0807 03:00:33.928664 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:33.928773 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":279,"threads_started":1}
I20260807 03:00:34.422428 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:34.421729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:59415)
0807 03:00:34.422160 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:34.422243 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":222,"threads_started":1}
I20260807 03:00:34.441205 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:34.440422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:45785)
0807 03:00:34.440953 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:34.441037 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":243,"threads_started":1}
I20260807 03:00:34.930713 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:34.929882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:53209)
0807 03:00:34.930459 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:34.930540 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":341,"threads_started":1}
I20260807 03:00:35.425581 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:35.423401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:44183)
0807 03:00:35.425252 (+ 1851us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:35.425369 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":249,"threads_started":1}
I20260807 03:00:35.442996 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:35.442118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:37983)
0807 03:00:35.442728 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:35.442819 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":307,"threads_started":1}
I20260807 03:00:35.932559 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:35.931751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:51913)
0807 03:00:35.932306 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:35.932388 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":322,"threads_started":1}
I20260807 03:00:36.427804 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:36.426826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:45497)
0807 03:00:36.427498 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:36.427624 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":409,"threads_started":1}
I20260807 03:00:36.444825 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:36.443943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:40951)
0807 03:00:36.444552 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:36.444639 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":377,"threads_started":1}
I20260807 03:00:36.934494 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:36.933577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:37419)
0807 03:00:36.934189 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:36.934306 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":275,"threads_started":1}
I20260807 03:00:37.429456 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:37.428699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:58313)
0807 03:00:37.429212 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:37.429304 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":286,"threads_started":1}
I20260807 03:00:37.446820 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:37.445900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:45229)
0807 03:00:37.446524 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:37.446607 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":308,"threads_started":1}
I20260807 03:00:37.936306 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:37.935447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:34433)
0807 03:00:37.935973 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:37.936069 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":231,"threads_started":1}
I20260807 03:00:38.431385 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:38.430554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:46837)
0807 03:00:38.431066 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:38.431158 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":233,"threads_started":1}
I20260807 03:00:38.448499 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:38.447759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:43955)
0807 03:00:38.448248 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:38.448335 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
I20260807 03:00:38.938427 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:38.937492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:37687)
0807 03:00:38.938115 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:38.938228 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":278,"threads_started":1}
I20260807 03:00:39.433118 25797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:39.432325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:44917)
0807 03:00:39.432789 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:39.432942 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":223,"threads_started":1}
I20260807 03:00:39.450297 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:39.449461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:47987)
0807 03:00:39.450015 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:39.450106 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":310,"threads_started":1}
I20260807 03:00:39.940372 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:39.939488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:45049)
0807 03:00:39.940069 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:39.940198 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":313,"threads_started":1}
I20260807 03:00:40.434703 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:40.433975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:56459)
0807 03:00:40.434443 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:40.434522 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260807 03:00:40.451941 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:40.451248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:54911)
0807 03:00:40.451694 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:40.451781 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
I20260807 03:00:40.942307 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:40.941519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:45709)
0807 03:00:40.942029 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:40.942124 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":271,"threads_started":1}
I20260807 03:00:41.436448 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:41.435727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:52075)
0807 03:00:41.436201 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:41.436287 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":236,"threads_started":1}
I20260807 03:00:41.453809 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:41.452865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:35391)
0807 03:00:41.453443 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:41.453537 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":331,"threads_started":1}
I20260807 03:00:41.944234 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:41.943325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:34881)
0807 03:00:41.943878 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:41.943991 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":292,"threads_started":1}
I20260807 03:00:42.438189 25808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:42.437464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:36721)
0807 03:00:42.437899 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:42.437994 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260807 03:00:42.455541 25809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:42.454710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:54435)
0807 03:00:42.455201 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:42.455312 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260807 03:00:42.946468 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:42.945689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:55217)
0807 03:00:42.946213 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:42.946305 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":255,"threads_started":1}
I20260807 03:00:43.440021 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:43.439232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:52153)
0807 03:00:43.439773 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:43.439862 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260807 03:00:43.457396 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:43.456517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:59463)
0807 03:00:43.457082 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:43.457200 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":259,"threads_started":1}
I20260807 03:00:43.948441 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:43.947587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:60097)
0807 03:00:43.948126 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:43.948241 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":258,"threads_started":1}
I20260807 03:00:44.441911 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:44.441064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:51881)
0807 03:00:44.441617 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:44.441716 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":256,"threads_started":1}
I20260807 03:00:44.459002 25817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:44.458280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:56423)
0807 03:00:44.458739 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:44.458822 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260807 03:00:44.950114 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:44.949419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:43855)
0807 03:00:44.949847 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:44.949931 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":211,"threads_started":1}
I20260807 03:00:45.443717 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:45.442872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:54799)
0807 03:00:45.443385 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:45.443470 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":305,"threads_started":1}
I20260807 03:00:45.461009 25820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:45.460085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.65:45103)
0807 03:00:45.460661 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:45.460753 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":298,"threads_started":1}
I20260807 03:00:45.952065 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:45.951307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.66:35703)
0807 03:00:45.951775 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:45.951873 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":233,"threads_started":1}
I20260807 03:00:46.422381 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 25149
I20260807 03:00:46.445938 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:46.445039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:37333 (local address 127.1.121.67:56153)
0807 03:00:46.445595 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:46.445721 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.126:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260807 03:00:46.448688 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:46.423614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.1.121.65:39757)
0807 03:00:46.424496 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:46.424519 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:46.424569 (+ 50us) server_negotiation.cc:407] Connection header received
0807 03:00:46.427297 (+ 2728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:46.427312 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:46.427905 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:46.428310 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:46.429936 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:46.431377 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:46.432686 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:46.433440 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:46.433583 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:46.441473 (+ 7890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:46.441483 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:46.441503 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:46.441553 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:46.444829 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:46.445779 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:46.445789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:46.445800 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:46.446043 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:46.446489 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:46.446493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:46.446495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:46.447144 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:46.447401 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:46.447793 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 03:00:46.448373 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":460,"threads_started":1}
I20260807 03:00:46.450198 25398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:46.537132 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 25149
I20260807 03:00:46.564242 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 25406
I20260807 03:00:46.582523 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:46.565354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.1.121.66:39309)
0807 03:00:46.565879 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:46.565893 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:46.565932 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:00:46.567563 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:46.567571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:46.567980 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:46.568274 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:46.569714 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:46.571027 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:46.572272 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:46.572987 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:46.573122 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:46.575798 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:46.575806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:46.575822 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:46.575859 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:46.579100 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:46.579917 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:46.579924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:46.579931 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:46.580109 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:46.580496 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:46.580499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:46.580501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:46.581097 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:46.581360 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:46.581758 (+ 398us) server_negotiation.cc:299] Negotiation successful
0807 03:00:46.582216 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":248,"threads_started":1}
I20260807 03:00:46.584169 25579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:46.674314 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 25406
I20260807 03:00:46.703365 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 25589
I20260807 03:00:46.723416 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:46.704658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.1.121.67:38517)
0807 03:00:46.705372 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:46.705386 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:46.705430 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:00:46.707291 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:46.707301 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:46.707757 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:46.708136 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:46.709900 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:46.711240 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:46.712666 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:46.713442 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:46.713709 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:46.716685 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:46.716694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:46.716711 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:46.716755 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:46.719735 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:46.720498 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:46.720506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:46.720513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:46.720703 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:46.721239 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:46.721246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:46.721250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:46.721821 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:46.722152 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:46.722632 (+ 480us) server_negotiation.cc:299] Negotiation successful
0807 03:00:46.723108 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":375,"threads_started":1}
I20260807 03:00:46.724960 25759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:46.809821 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 25589
2026-08-07T03:00:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65331 ms)
[----------] 30 tests from SecurityITest (323105 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:00:46 dist-test-slave-3h5h krb5kdc[25838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:00:46 dist-test-slave-3h5h krb5kdc[25838](info): set up 2 sockets
Aug 07 03:00:46 dist-test-slave-3h5h krb5kdc[25838](info): commencing operation
krb5kdc: starting...
W20260807 03:00:48.161137 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.007s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:00:48 dist-test-slave-3h5h krb5kdc[25838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:00:48.497012 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34569
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:34569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:48.855002 25856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:48.855315 25856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:48.855445 25856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:48.862898 25856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:00:48.863049 25856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:48.863143 25856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:48.863247 25856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:00:48.863332 25856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:00:48.873766 25856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:34569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:48.875165 25856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:48.876529 25856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:48.889592 25865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:48.889763 25872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:48.889737 25867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:48.890265 25856 server_base.cc:1034] running on GCE node
I20260807 03:00:48.891119 25856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:48.892508 25856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:48.893734 25856 hybrid_clock.cc:648] HybridClock initialized: now 1786071648893682 us; error 34 us; skew 500 ppm
I20260807 03:00:48.894129 25856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:48 dist-test-slave-3h5h krb5kdc[25838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071648, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:48.897333 25856 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:00:48.898999 25856 webserver.cc:460] Webserver started at http://127.1.121.126:42049/ using document root <none> and password file <none>
I20260807 03:00:48.899730 25856 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:48.899865 25856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:48.900200 25856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:48.903371 25856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "5c9457f9680c4d9482b7a9fc23cc4762"
format_stamp: "Formatted at 2026-08-07 03:00:48 on dist-test-slave-3h5h"
server_key: "767914e69b0df91430d78bab650979e6"
server_key_iv: "2c00da3b7bfa873536af1af5038c5a20"
server_key_version: "encryptionkey@0"
I20260807 03:00:48.904165 25856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "5c9457f9680c4d9482b7a9fc23cc4762"
format_stamp: "Formatted at 2026-08-07 03:00:48 on dist-test-slave-3h5h"
server_key: "767914e69b0df91430d78bab650979e6"
server_key_iv: "2c00da3b7bfa873536af1af5038c5a20"
server_key_version: "encryptionkey@0"
I20260807 03:00:48.909502 25856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260807 03:00:48.913060 25884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:48.914438 25856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:00:48.914635 25856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "5c9457f9680c4d9482b7a9fc23cc4762"
format_stamp: "Formatted at 2026-08-07 03:00:48 on dist-test-slave-3h5h"
server_key: "767914e69b0df91430d78bab650979e6"
server_key_iv: "2c00da3b7bfa873536af1af5038c5a20"
server_key_version: "encryptionkey@0"
I20260807 03:00:48.914855 25856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:48.934252 25856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:48.938493 25856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:48.938802 25856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:48.958683 25856 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:34569
I20260807 03:00:48.958701 25956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:34569 every 8 connection(s)
I20260807 03:00:48.960250 25856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:00:48.964207 25958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:48.970361 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 25856
I20260807 03:00:48.970588 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:00:48.971248 1509 external_mini_cluster.cc:1428] Setting key 5c533eccb127d33e1afda1814f2353cc
I20260807 03:00:48.975636 25958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762: Bootstrap starting.
I20260807 03:00:48.979660 25958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:48.981109 25958 log.cc:824] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:48.984576 25958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762: No bootstrap required, opened a new log
Aug 07 03:00:48 dist-test-slave-3h5h krb5kdc[25838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:48.994640 25958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9457f9680c4d9482b7a9fc23cc4762" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34569 } }
I20260807 03:00:48.995157 25958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:48.995291 25958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9457f9680c4d9482b7a9fc23cc4762, State: Initialized, Role: FOLLOWER
I20260807 03:00:48.995898 25958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9457f9680c4d9482b7a9fc23cc4762" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34569 } }
I20260807 03:00:48.996160 25958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:00:48.996351 25958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:00:48.996548 25958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:48.998498 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:48.973391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.1.121.126:34569)
0807 03:00:48.974266 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:48.974283 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:48.974335 (+ 52us) server_negotiation.cc:407] Connection header received
0807 03:00:48.976873 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:48.976906 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:48.977420 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:48.977932 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:48.979522 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:48.981672 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:48.983393 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:48.984425 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:48.984622 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:48.989412 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:48.989428 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:48.989449 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:48.989501 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:48.993737 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:48.994555 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:48.994564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:48.994573 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:48.994834 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:48.995400 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:48.995407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:48.995410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:48.996086 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:48.996430 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:48.996984 (+ 554us) server_negotiation.cc:299] Negotiation successful
0807 03:00:48.997426 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":375,"threads_started":1}
I20260807 03:00:48.999282 25958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9457f9680c4d9482b7a9fc23cc4762" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34569 } }
I20260807 03:00:48.999794 25958 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c9457f9680c4d9482b7a9fc23cc4762; no voters:
I20260807 03:00:49.000491 25958 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:00:49.000620 25963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:00:49.002590 25958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:00:49.002849 25963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [term 1 LEADER]: Becoming Leader. State: Replica: 5c9457f9680c4d9482b7a9fc23cc4762, State: Running, Role: LEADER
I20260807 03:00:49.003356 25963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9457f9680c4d9482b7a9fc23cc4762" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34569 } }
I20260807 03:00:49.006716 25964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c9457f9680c4d9482b7a9fc23cc4762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9457f9680c4d9482b7a9fc23cc4762" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34569 } } }
I20260807 03:00:49.007609 25964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [sys.catalog]: This master's current role is: LEADER
I20260807 03:00:49.006728 25965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9457f9680c4d9482b7a9fc23cc4762. Latest consensus state: current_term: 1 leader_uuid: "5c9457f9680c4d9482b7a9fc23cc4762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9457f9680c4d9482b7a9fc23cc4762" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34569 } } }
I20260807 03:00:49.007848 25965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762 [sys.catalog]: This master's current role is: LEADER
I20260807 03:00:49.009464 25976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:00:49.014365 25976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:00:49.023567 25976 catalog_manager.cc:1347] Generated new cluster ID: 18e220feb26e4582aa5252968199d938
I20260807 03:00:49.023734 25976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:00:49.044613 25976 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:00:49.046712 25976 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:00:49.055117 25976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762: Generated new TSK 0
I20260807 03:00:49.056041 25976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 03:00:49.157423 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34569
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:49.524497 25987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:49.524796 25987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:49.524972 25987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:49.535102 25987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:49.535246 25987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:49.535424 25987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 03:00:49.545876 25987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:49.547227 25987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:49.548583 25987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:49.563207 25992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:49.563292 25997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:49.563238 25994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:49.563956 25987 server_base.cc:1034] running on GCE node
I20260807 03:00:49.564812 25987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:49.565858 25987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:49.567111 25987 hybrid_clock.cc:648] HybridClock initialized: now 1786071649567063 us; error 33 us; skew 500 ppm
I20260807 03:00:49.567502 25987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:49 dist-test-slave-3h5h krb5kdc[25838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071649, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:49.571337 25987 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 03:00:49.572912 25987 webserver.cc:460] Webserver started at http://127.1.121.65:37573/ using document root <none> and password file <none>
I20260807 03:00:49.573594 25987 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:49.573719 25987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:49.573990 25987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:49.576987 25987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "5f6e43004f6942b2965f5773c2347946"
format_stamp: "Formatted at 2026-08-07 03:00:49 on dist-test-slave-3h5h"
server_key: "a2d39f41d3cdcfbd65883ade5f46b7e1"
server_key_iv: "ffd5777d504f2ef71a426578a8ff8d27"
server_key_version: "encryptionkey@0"
I20260807 03:00:49.577765 25987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "5f6e43004f6942b2965f5773c2347946"
format_stamp: "Formatted at 2026-08-07 03:00:49 on dist-test-slave-3h5h"
server_key: "a2d39f41d3cdcfbd65883ade5f46b7e1"
server_key_iv: "ffd5777d504f2ef71a426578a8ff8d27"
server_key_version: "encryptionkey@0"
I20260807 03:00:49.583192 25987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260807 03:00:49.586793 26010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:49.588208 25987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:00:49.588347 25987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "5f6e43004f6942b2965f5773c2347946"
format_stamp: "Formatted at 2026-08-07 03:00:49 on dist-test-slave-3h5h"
server_key: "a2d39f41d3cdcfbd65883ade5f46b7e1"
server_key_iv: "ffd5777d504f2ef71a426578a8ff8d27"
server_key_version: "encryptionkey@0"
I20260807 03:00:49.588521 25987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:49.602807 25987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:49.607187 25987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:49.607440 25987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:49.608364 25987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:49.609767 25987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:49.609843 25987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:49.609931 25987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:49.609982 25987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:49.644678 25987 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:46487
I20260807 03:00:49.644846 26170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:46487 every 8 connection(s)
I20260807 03:00:49.646067 25987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 03:00:49.648527 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 25987
I20260807 03:00:49.648764 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 03:00:49.649681 1509 external_mini_cluster.cc:1428] Setting key 88f9b56bf9e7e5974fa210f4756c9dcb
Aug 07 03:00:49 dist-test-slave-3h5h krb5kdc[25838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071649, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:49.671209 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:49.648907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:59419 (local address 127.1.121.126:34569)
0807 03:00:49.649446 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:49.649452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:49.650580 (+ 1128us) server_negotiation.cc:407] Connection header received
0807 03:00:49.653091 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:49.653095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:49.653164 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:49.653258 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:49.655539 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:49.656271 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:49.658818 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:49.659461 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:49.659594 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:49.664364 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:49.664374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:49.664382 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:49.664430 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:49.667646 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:49.668676 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:49.668680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:49.668683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:49.668807 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:49.669366 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:49.669369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:49.669371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:49.669682 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:49.669800 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:49.670774 (+ 974us) server_negotiation.cc:299] Negotiation successful
0807 03:00:49.670950 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
I20260807 03:00:49.672900 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:49.649433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:34569 (local address 127.1.121.65:59419)
0807 03:00:49.650366 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:49.650433 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:49.652749 (+ 2316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:49.653553 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:49.653566 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:49.654448 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:49.655336 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:49.655366 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:49.656456 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:49.656465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:49.658639 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:49.658658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:49.659723 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:49.659733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:49.660083 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:49.660930 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:00:49.660968 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:49.664079 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:49.667819 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:49.667831 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:49.667852 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:49.668512 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:49.668963 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:49.668971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:49.668977 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:49.669245 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:49.669821 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:49.669832 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:49.670340 (+ 508us) client_negotiation.cc:769] Sending connection context
0807 03:00:49.670685 (+ 345us) client_negotiation.cc:240] Negotiation successful
0807 03:00:49.671337 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":356,"threads_started":1}
I20260807 03:00:49.674605 26171 heartbeater.cc:347] Connected to a master server at 127.1.121.126:34569
I20260807 03:00:49.674945 26171 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:49.675957 26171 heartbeater.cc:511] Master 127.1.121.126:34569 requested a full tablet report, sending...
I20260807 03:00:49.678237 25906 ts_manager.cc:194] Registered new tserver with Master: 5f6e43004f6942b2965f5773c2347946 (127.1.121.65:46487)
I20260807 03:00:49.679706 25906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:59419
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 03:00:49.728411 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34569
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:50.083972 26179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:50.084321 26179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:50.084491 26179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:50.092286 26179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:50.092442 26179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:50.092621 26179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 03:00:50.102885 26179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:50.104331 26179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:50.105775 26179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:50.119935 26187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:50.119869 26190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:50.119976 26186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:50.120910 26179 server_base.cc:1034] running on GCE node
I20260807 03:00:50.121579 26179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:50.122632 26179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:50.123868 26179 hybrid_clock.cc:648] HybridClock initialized: now 1786071650123814 us; error 47 us; skew 500 ppm
I20260807 03:00:50.124209 26179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:50 dist-test-slave-3h5h krb5kdc[25838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071650, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:50.128264 26179 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 03:00:50.131418 26179 webserver.cc:460] Webserver started at http://127.1.121.66:32857/ using document root <none> and password file <none>
I20260807 03:00:50.132452 26179 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:50.132604 26179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:50.132985 26179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:50.136966 26179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "df9bc41685b440de871cd6c501df0c40"
format_stamp: "Formatted at 2026-08-07 03:00:50 on dist-test-slave-3h5h"
server_key: "0805c36dfeb793d442516b326d939f1e"
server_key_iv: "b10f2e576519738898de066349f218a5"
server_key_version: "encryptionkey@0"
I20260807 03:00:50.138103 26179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "df9bc41685b440de871cd6c501df0c40"
format_stamp: "Formatted at 2026-08-07 03:00:50 on dist-test-slave-3h5h"
server_key: "0805c36dfeb793d442516b326d939f1e"
server_key_iv: "b10f2e576519738898de066349f218a5"
server_key_version: "encryptionkey@0"
I20260807 03:00:50.143872 26179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 03:00:50.147729 26198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:50.149338 26179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:00:50.149546 26179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "df9bc41685b440de871cd6c501df0c40"
format_stamp: "Formatted at 2026-08-07 03:00:50 on dist-test-slave-3h5h"
server_key: "0805c36dfeb793d442516b326d939f1e"
server_key_iv: "b10f2e576519738898de066349f218a5"
server_key_version: "encryptionkey@0"
I20260807 03:00:50.149802 26179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:50.177982 26179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:50.182704 26179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:50.183022 26179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:50.184049 26179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:50.185528 26179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:50.185639 26179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:50.185791 26179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:50.185866 26179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:50.221199 26179 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:44999
I20260807 03:00:50.221251 26362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:44999 every 8 connection(s)
I20260807 03:00:50.222575 26179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 03:00:50.223832 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 26179
I20260807 03:00:50.224023 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 03:00:50.224614 1509 external_mini_cluster.cc:1428] Setting key 222fe947d49db9fe687b411847b9b534
Aug 07 03:00:50 dist-test-slave-3h5h krb5kdc[25838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071650, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:50.248225 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.225617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:47465 (local address 127.1.121.126:34569)
0807 03:00:50.226178 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:50.226183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:50.227379 (+ 1196us) server_negotiation.cc:407] Connection header received
0807 03:00:50.230414 (+ 3035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:50.230418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:50.230487 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:50.230586 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:50.232920 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.233794 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.236296 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.236801 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.236992 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.241395 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:50.241400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:50.241403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:50.241438 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:50.244748 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:50.245584 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:50.245592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:50.245598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:50.245778 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:50.246213 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:50.246221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:50.246227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:50.246619 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:50.246766 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:50.247440 (+ 674us) server_negotiation.cc:299] Negotiation successful
0807 03:00:50.247694 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":326,"threads_started":1}
I20260807 03:00:50.249046 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.226253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:34569 (local address 127.1.121.66:47465)
0807 03:00:50.227183 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:50.227234 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:50.230078 (+ 2844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:50.230879 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:50.230893 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:50.231717 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:50.232555 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.232582 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.233970 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.233978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.236105 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.236130 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.237131 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.237141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.237481 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.238263 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:00:50.238297 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:50.241145 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:50.244944 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:50.244951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:50.244965 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:50.245455 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:50.245904 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:50.245907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:50.245909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:50.246091 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:50.246752 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:50.246757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:50.247067 (+ 310us) client_negotiation.cc:769] Sending connection context
0807 03:00:50.247308 (+ 241us) client_negotiation.cc:240] Negotiation successful
0807 03:00:50.247719 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":408,"threads_started":1}
I20260807 03:00:50.250885 26364 heartbeater.cc:347] Connected to a master server at 127.1.121.126:34569
I20260807 03:00:50.251298 26364 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:50.252218 26364 heartbeater.cc:511] Master 127.1.121.126:34569 requested a full tablet report, sending...
I20260807 03:00:50.254158 25906 ts_manager.cc:194] Registered new tserver with Master: df9bc41685b440de871cd6c501df0c40 (127.1.121.66:44999)
I20260807 03:00:50.255533 25906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:47465
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 03:00:50.301930 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34569
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:50.658200 26373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:50.658591 26373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:50.658766 26373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:50.666174 26373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:50.666330 26373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:50.666522 26373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 03:00:50.676434 26373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:50.677778 26373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:50.679136 26373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 03:00:50.683691 26171 heartbeater.cc:503] Master 127.1.121.126:34569 was elected leader, sending a full tablet report...
W20260807 03:00:50.694101 26379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:50.694021 26383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:50.694096 26380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:50.695004 26373 server_base.cc:1034] running on GCE node
I20260807 03:00:50.695580 26373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:50.696470 26373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:50.697682 26373 hybrid_clock.cc:648] HybridClock initialized: now 1786071650697601 us; error 70 us; skew 500 ppm
I20260807 03:00:50.697980 26373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:50 dist-test-slave-3h5h krb5kdc[25838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071650, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:50.701277 26373 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 03:00:50.702816 26373 webserver.cc:460] Webserver started at http://127.1.121.67:46609/ using document root <none> and password file <none>
I20260807 03:00:50.703516 26373 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:50.703624 26373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:50.703949 26373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:50.707059 26373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "9a469e5b203d459d8fb56fd4e52ddb8a"
format_stamp: "Formatted at 2026-08-07 03:00:50 on dist-test-slave-3h5h"
server_key: "90afbef4d68cd7bfaf0f1b88e055ab4d"
server_key_iv: "cc17d8b62742018f9f487d858039c674"
server_key_version: "encryptionkey@0"
I20260807 03:00:50.707861 26373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "9a469e5b203d459d8fb56fd4e52ddb8a"
format_stamp: "Formatted at 2026-08-07 03:00:50 on dist-test-slave-3h5h"
server_key: "90afbef4d68cd7bfaf0f1b88e055ab4d"
server_key_iv: "cc17d8b62742018f9f487d858039c674"
server_key_version: "encryptionkey@0"
I20260807 03:00:50.714272 26373 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260807 03:00:50.717824 26392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:50.719200 26373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:00:50.719380 26373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "9a469e5b203d459d8fb56fd4e52ddb8a"
format_stamp: "Formatted at 2026-08-07 03:00:50 on dist-test-slave-3h5h"
server_key: "90afbef4d68cd7bfaf0f1b88e055ab4d"
server_key_iv: "cc17d8b62742018f9f487d858039c674"
server_key_version: "encryptionkey@0"
I20260807 03:00:50.719585 26373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:50.735734 26373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:50.740325 26373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:50.740617 26373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:50.741648 26373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:50.742980 26373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:50.743053 26373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:50.743232 26373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:50.743290 26373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:50.777629 26373 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:34575
I20260807 03:00:50.777691 26558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:34575 every 8 connection(s)
I20260807 03:00:50.779019 26373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 03:00:50.784775 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 26373
I20260807 03:00:50.784997 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 03:00:50.785559 1509 external_mini_cluster.cc:1428] Setting key ba8594defca6fd95852531a2ca7f8167
Aug 07 03:00:50 dist-test-slave-3h5h krb5kdc[25838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071650, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:50.800714 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.782198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:34931 (local address 127.1.121.126:34569)
0807 03:00:50.782720 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:50.782727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:50.783606 (+ 879us) server_negotiation.cc:407] Connection header received
0807 03:00:50.786016 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:50.786021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:50.786104 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:50.786253 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:50.788320 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.789436 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.791050 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.791495 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.791589 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.795121 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:50.795126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:50.795129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:50.795159 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:50.797679 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:50.798354 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:50.798358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:50.798360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:50.798498 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:50.798861 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:50.798864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:50.798866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:50.799146 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:50.799302 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:50.800079 (+ 777us) server_negotiation.cc:299] Negotiation successful
0807 03:00:50.800288 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":242,"threads_started":1}
I20260807 03:00:50.801776 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.782646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:34569 (local address 127.1.121.67:34931)
0807 03:00:50.783411 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:50.783456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:50.785675 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:50.786560 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:50.786572 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:50.787351 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:50.788151 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.788171 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.789605 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.789609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.790915 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.790927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.791680 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.791685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.791973 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.792662 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:00:50.792684 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:50.794939 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:50.797803 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:50.797810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:50.797821 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:50.798244 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:50.798593 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:50.798597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:50.798599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:50.798769 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:50.799256 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:50.799262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:50.799591 (+ 329us) client_negotiation.cc:769] Sending connection context
0807 03:00:50.799906 (+ 315us) client_negotiation.cc:240] Negotiation successful
0807 03:00:50.800437 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":295,"threads_started":1}
I20260807 03:00:50.803107 26559 heartbeater.cc:347] Connected to a master server at 127.1.121.126:34569
I20260807 03:00:50.803432 26559 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:50.804149 26559 heartbeater.cc:511] Master 127.1.121.126:34569 requested a full tablet report, sending...
I20260807 03:00:50.805580 25906 ts_manager.cc:194] Registered new tserver with Master: 9a469e5b203d459d8fb56fd4e52ddb8a (127.1.121.67:34575)
I20260807 03:00:50.806419 25906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:34931
I20260807 03:00:50.814028 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:00:50.831616 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.818723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.1.121.126:34569)
0807 03:00:50.818952 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:50.818960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:50.819529 (+ 569us) server_negotiation.cc:407] Connection header received
0807 03:00:50.819682 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:50.819686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:50.819750 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:50.819835 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:50.821479 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.822207 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.823615 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.824155 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.824271 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.826675 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:50.826680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:50.826683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:50.826715 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:50.829417 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:50.830016 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:50.830020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:50.830022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:50.830143 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:50.830463 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:50.830466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:50.830468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:50.830776 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:50.830977 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:50.831235 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 03:00:50.831396 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:00:50.835855 25906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:00:50.842904 25906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:00:50.867514 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.855923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:46487 (local address 127.0.0.1:59050)
0807 03:00:50.857986 (+ 2063us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:50.858008 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:50.858233 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:50.859845 (+ 1612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:50.859849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:50.859874 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:50.860465 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.860479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.862339 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.862350 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.864783 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.864805 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.866378 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.866384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.866583 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.866640 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 03:00:50.866796 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 03:00:50.867010 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1917,"mutex_wait_us":56,"thread_start_us":608,"threads_started":1}
I20260807 03:00:50.868137 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.855864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.1.121.66:44999)
0807 03:00:50.856583 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:50.856592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:50.858046 (+ 1454us) server_negotiation.cc:407] Connection header received
0807 03:00:50.858519 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:50.858532 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:50.858848 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:50.859025 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:50.860320 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.861935 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.865099 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.866420 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.866572 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.866749 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:50.867325 (+ 576us) server_negotiation.cc:299] Negotiation successful
0807 03:00:50.867515 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":330,"threads_started":1}
I20260807 03:00:50.868324 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.856272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.1.121.65:46487)
0807 03:00:50.859406 (+ 3134us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:50.859411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:50.859433 (+ 22us) server_negotiation.cc:407] Connection header received
0807 03:00:50.859505 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:50.859509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:50.859659 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:50.859854 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:50.860723 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.862189 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.865008 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.866180 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.866349 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.866601 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:50.866854 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 03:00:50.867018 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2965,"thread_start_us":2827,"threads_started":1}
I20260807 03:00:50.868413 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.857710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.1.121.67:34575)
0807 03:00:50.858924 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:50.858931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:50.860651 (+ 1720us) server_negotiation.cc:407] Connection header received
0807 03:00:50.861002 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:50.861011 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:50.861234 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:50.861389 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:50.862042 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.863134 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.865570 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.866987 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.867106 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.867339 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:50.867901 (+ 562us) server_negotiation.cc:299] Negotiation successful
0807 03:00:50.868061 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":898,"threads_started":1}
I20260807 03:00:50.869048 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.855699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:44999 (local address 127.0.0.1:56706)
0807 03:00:50.857838 (+ 2139us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:50.857891 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:50.858235 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:50.859451 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:50.859462 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:50.859508 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:50.860044 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.860065 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.862051 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.862056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.864868 (+ 2812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.864883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.866904 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.866913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.867100 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.867134 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 03:00:50.868437 (+ 1303us) client_negotiation.cc:240] Negotiation successful
0807 03:00:50.868585 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":55,"spinlock_wait_cycles":4864,"thread_start_us":4522,"threads_started":1}
I20260807 03:00:50.869148 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.857353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:34575 (local address 127.0.0.1:45692)
0807 03:00:50.860537 (+ 3184us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:50.860556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:50.860773 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:50.861496 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:50.861501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:50.861525 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:50.861861 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.861875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.863287 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.863293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.865394 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.865409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.867516 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.867522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.867709 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.867739 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 03:00:50.868645 (+ 906us) client_negotiation.cc:240] Negotiation successful
0807 03:00:50.868793 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3052,"thread_start_us":320,"threads_started":1}
I20260807 03:00:50.872256 26074 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff5bb40aaeb48de88cba490cd4438bd (DEFAULT_TABLE table=test-table [id=39e9271d57024ac6b8efe854dfcb540b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:50.872586 26275 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff5bb40aaeb48de88cba490cd4438bd (DEFAULT_TABLE table=test-table [id=39e9271d57024ac6b8efe854dfcb540b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:50.872586 26464 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff5bb40aaeb48de88cba490cd4438bd (DEFAULT_TABLE table=test-table [id=39e9271d57024ac6b8efe854dfcb540b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:50.873761 26074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff5bb40aaeb48de88cba490cd4438bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:50.873998 26464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff5bb40aaeb48de88cba490cd4438bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:50.874001 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff5bb40aaeb48de88cba490cd4438bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:50.885725 26591 tablet_bootstrap.cc:492] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: Bootstrap starting.
I20260807 03:00:50.888684 26592 tablet_bootstrap.cc:492] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: Bootstrap starting.
I20260807 03:00:50.891026 26591 tablet_bootstrap.cc:654] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:50.892349 26592 tablet_bootstrap.cc:654] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:50.892433 26591 log.cc:824] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:50.893558 26594 tablet_bootstrap.cc:492] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: Bootstrap starting.
I20260807 03:00:50.893807 26592 log.cc:824] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:50.896802 26594 tablet_bootstrap.cc:654] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:50.897604 26592 tablet_bootstrap.cc:492] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: No bootstrap required, opened a new log
I20260807 03:00:50.897953 26592 ts_tablet_manager.cc:1397] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 03:00:50.898192 26594 log.cc:824] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:50.899762 26591 tablet_bootstrap.cc:492] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: No bootstrap required, opened a new log
I20260807 03:00:50.900038 26591 ts_tablet_manager.cc:1397] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260807 03:00:50.908604 26594 tablet_bootstrap.cc:492] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: No bootstrap required, opened a new log
I20260807 03:00:50.908946 26594 ts_tablet_manager.cc:1397] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260807 03:00:50.909158 26592 raft_consensus.cc:348] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } }
I20260807 03:00:50.909637 26592 raft_consensus.cc:374] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:50.909749 26592 raft_consensus.cc:729] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a469e5b203d459d8fb56fd4e52ddb8a, State: Initialized, Role: FOLLOWER
I20260807 03:00:50.910499 26592 consensus_queue.cc:260] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } }
I20260807 03:00:50.910887 26591 raft_consensus.cc:348] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } }
I20260807 03:00:50.911361 26591 raft_consensus.cc:374] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:50.911465 26591 raft_consensus.cc:729] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6e43004f6942b2965f5773c2347946, State: Initialized, Role: FOLLOWER
I20260807 03:00:50.911911 26559 heartbeater.cc:503] Master 127.1.121.126:34569 was elected leader, sending a full tablet report...
I20260807 03:00:50.912410 26592 ts_tablet_manager.cc:1428] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260807 03:00:50.913023 26591 consensus_queue.cc:260] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } }
I20260807 03:00:50.915082 26591 ts_tablet_manager.cc:1428] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: Time spent starting tablet: real 0.015s user 0.005s sys 0.010s
I20260807 03:00:50.919678 26594 raft_consensus.cc:348] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } }
I20260807 03:00:50.920011 26594 raft_consensus.cc:374] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:50.920082 26594 raft_consensus.cc:729] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9bc41685b440de871cd6c501df0c40, State: Initialized, Role: FOLLOWER
I20260807 03:00:50.920634 26594 consensus_queue.cc:260] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } }
I20260807 03:00:50.921690 26364 heartbeater.cc:503] Master 127.1.121.126:34569 was elected leader, sending a full tablet report...
I20260807 03:00:50.922144 26594 ts_tablet_manager.cc:1428] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260807 03:00:50.943316 26600 raft_consensus.cc:482] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:00:50.943668 26600 raft_consensus.cc:504] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } }
I20260807 03:00:50.945617 26600 leader_election.cc:288] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df9bc41685b440de871cd6c501df0c40 (127.1.121.66:44999), 9a469e5b203d459d8fb56fd4e52ddb8a (127.1.121.67:34575)
I20260807 03:00:50.952646 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.946235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:32945 (local address 127.1.121.67:34575)
0807 03:00:50.946475 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:50.946484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:50.947242 (+ 758us) server_negotiation.cc:407] Connection header received
0807 03:00:50.947450 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:50.947454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:50.947514 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:50.947613 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:50.948339 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.949155 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.950907 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.951672 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.951864 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.951948 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:50.952250 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 03:00:50.952380 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 03:00:50.952711 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.945977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:51991 (local address 127.1.121.66:44999)
0807 03:00:50.946214 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:50.946221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:50.947162 (+ 941us) server_negotiation.cc:407] Connection header received
0807 03:00:50.947391 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:50.947396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:50.947460 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:50.947557 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:50.948264 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.949039 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.950927 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.951692 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.951898 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.951998 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:50.952314 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 03:00:50.952448 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 03:00:50.952766 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.946123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:34575 (local address 127.1.121.65:32945)
0807 03:00:50.947108 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:50.947122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:50.947286 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:50.947629 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:50.947634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:50.947657 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:50.948133 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.948143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.949291 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.949298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.950748 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.950759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.951914 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.951921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.952071 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.952098 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 03:00:50.952208 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 03:00:50.952336 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":22,"thread_start_us":561,"threads_started":1}
I20260807 03:00:50.953112 26603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.945996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:44999 (local address 127.1.121.65:51991)
0807 03:00:50.947038 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:50.947059 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:50.947231 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:50.947576 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:50.947580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:50.947603 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:50.948102 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.948115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.949165 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.949170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.950753 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:50.950766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.951998 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.952005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:50.952150 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.952175 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 03:00:50.952283 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 03:00:50.952408 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":78,"thread_start_us":501,"threads_started":1}
I20260807 03:00:50.954164 26302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff5bb40aaeb48de88cba490cd4438bd" candidate_uuid: "5f6e43004f6942b2965f5773c2347946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df9bc41685b440de871cd6c501df0c40" is_pre_election: true
I20260807 03:00:50.954154 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff5bb40aaeb48de88cba490cd4438bd" candidate_uuid: "5f6e43004f6942b2965f5773c2347946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" is_pre_election: true
I20260807 03:00:50.954767 26493 raft_consensus.cc:2454] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f6e43004f6942b2965f5773c2347946 in term 0.
I20260807 03:00:50.954761 26302 raft_consensus.cc:2454] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f6e43004f6942b2965f5773c2347946 in term 0.
I20260807 03:00:50.955516 26015 leader_election.cc:302] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f6e43004f6942b2965f5773c2347946, 9a469e5b203d459d8fb56fd4e52ddb8a; no voters:
I20260807 03:00:50.955962 26600 raft_consensus.cc:2781] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:00:50.956132 26600 raft_consensus.cc:482] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:00:50.956259 26600 raft_consensus.cc:3037] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:50.958633 26600 raft_consensus.cc:504] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } }
I20260807 03:00:50.959409 26600 leader_election.cc:288] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [CANDIDATE]: Term 1 election: Requested vote from peers df9bc41685b440de871cd6c501df0c40 (127.1.121.66:44999), 9a469e5b203d459d8fb56fd4e52ddb8a (127.1.121.67:34575)
I20260807 03:00:50.960260 26302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff5bb40aaeb48de88cba490cd4438bd" candidate_uuid: "5f6e43004f6942b2965f5773c2347946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df9bc41685b440de871cd6c501df0c40"
I20260807 03:00:50.960512 26302 raft_consensus.cc:3037] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:50.960496 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff5bb40aaeb48de88cba490cd4438bd" candidate_uuid: "5f6e43004f6942b2965f5773c2347946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a"
I20260807 03:00:50.960691 26493 raft_consensus.cc:3037] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:50.962880 26302 raft_consensus.cc:2454] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f6e43004f6942b2965f5773c2347946 in term 1.
I20260807 03:00:50.963331 26011 leader_election.cc:302] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f6e43004f6942b2965f5773c2347946, df9bc41685b440de871cd6c501df0c40; no voters:
I20260807 03:00:50.963631 26600 raft_consensus.cc:2781] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:00:50.963907 26493 raft_consensus.cc:2454] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f6e43004f6942b2965f5773c2347946 in term 1.
I20260807 03:00:50.964401 26600 raft_consensus.cc:686] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 1 LEADER]: Becoming Leader. State: Replica: 5f6e43004f6942b2965f5773c2347946, State: Running, Role: LEADER
I20260807 03:00:50.964990 26600 consensus_queue.cc:237] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } }
I20260807 03:00:50.969528 25906 catalog_manager.cc:5495] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f6e43004f6942b2965f5773c2347946 (127.1.121.65). New cstate: current_term: 1 leader_uuid: "5f6e43004f6942b2965f5773c2347946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f6e43004f6942b2965f5773c2347946" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 46487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 } health_report { overall_health: UNKNOWN } } }
W20260807 03:00:50.974747 26365 tablet.cc:2369] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:00:50.996119 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:50.990868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.1.121.65:46487)
0807 03:00:50.991063 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:50.991067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:50.991085 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:00:50.991266 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:50.991270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:50.991347 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:50.991476 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:00:50.991878 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.992696 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.994118 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:50.994529 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:50.994622 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:50.995105 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:00:50.995610 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:00:50.995740 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:50.995840 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 03:00:50.995905 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:00:51.004909 26302 raft_consensus.cc:1261] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [term 1 FOLLOWER]: Refusing update from remote peer 5f6e43004f6942b2965f5773c2347946: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:00:51.004906 26493 raft_consensus.cc:1261] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [term 1 FOLLOWER]: Refusing update from remote peer 5f6e43004f6942b2965f5773c2347946: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:00:51.005947 26607 consensus_queue.cc:1035] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a469e5b203d459d8fb56fd4e52ddb8a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 34575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:00:51.006340 26600 consensus_queue.cc:1035] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df9bc41685b440de871cd6c501df0c40" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:00:51.036602 25906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:00:51.036917 25906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:00:51.040701 25906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c9457f9680c4d9482b7a9fc23cc4762: Sending DeleteTablet for 3 replicas of tablet 9ff5bb40aaeb48de88cba490cd4438bd
I20260807 03:00:51.042002 26074 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff5bb40aaeb48de88cba490cd4438bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:51 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:59050
I20260807 03:00:51.041981 26464 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff5bb40aaeb48de88cba490cd4438bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:51 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:45692
I20260807 03:00:51.042785 26275 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff5bb40aaeb48de88cba490cd4438bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:51 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:56706
I20260807 03:00:51.046734 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 25987
I20260807 03:00:51.067551 26629 tablet_replica.cc:330] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: stopping tablet replica
I20260807 03:00:51.068287 26629 raft_consensus.cc:2229] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:00:51.069214 26629 raft_consensus.cc:2258] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:51.071460 26627 tablet_replica.cc:330] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: stopping tablet replica
I20260807 03:00:51.071877 26627 raft_consensus.cc:2229] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:00:51.072329 26627 raft_consensus.cc:2258] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:51.075264 26625 tablet_replica.cc:330] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: stopping tablet replica
I20260807 03:00:51.075482 26627 ts_tablet_manager.cc:1905] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:51.075731 26625 raft_consensus.cc:2229] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:00:51.076483 26625 raft_consensus.cc:2258] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:51.078263 26629 ts_tablet_manager.cc:1905] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:51.078830 26625 ts_tablet_manager.cc:1905] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 03:00:51 dist-test-slave-3h5h krb5kdc[25838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 03:00:51.089357 26625 ts_tablet_manager.cc:1918] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:51.089627 26625 log.cc:1196] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/9ff5bb40aaeb48de88cba490cd4438bd
I20260807 03:00:51.090318 26625 ts_tablet_manager.cc:1939] T 9ff5bb40aaeb48de88cba490cd4438bd P 5f6e43004f6942b2965f5773c2347946: Deleting consensus metadata
I20260807 03:00:51.091210 26627 ts_tablet_manager.cc:1918] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:51.091388 26627 log.cc:1196] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/9ff5bb40aaeb48de88cba490cd4438bd
I20260807 03:00:51.092993 25886 catalog_manager.cc:4841] TS 5f6e43004f6942b2965f5773c2347946 (127.1.121.65:46487): tablet 9ff5bb40aaeb48de88cba490cd4438bd (table test-table [id=39e9271d57024ac6b8efe854dfcb540b]) successfully deleted
I20260807 03:00:51.093484 26629 ts_tablet_manager.cc:1918] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:51.093673 26629 log.cc:1196] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/9ff5bb40aaeb48de88cba490cd4438bd
I20260807 03:00:51.094296 26629 ts_tablet_manager.cc:1939] T 9ff5bb40aaeb48de88cba490cd4438bd P df9bc41685b440de871cd6c501df0c40: Deleting consensus metadata
I20260807 03:00:51.095988 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:51.047914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.1.121.65:46487)
0807 03:00:51.048108 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:51.048114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:51.070461 (+ 22347us) server_negotiation.cc:407] Connection header received
0807 03:00:51.070687 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:51.070693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:51.070791 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:51.070925 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:51.073104 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:51.074556 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:51.077162 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:51.078005 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:51.078128 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:51.085289 (+ 7161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:51.085308 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:51.085330 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:51.085404 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:51.091196 (+ 5792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:51.092670 (+ 1474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:51.092683 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:51.092691 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:51.092997 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:51.093802 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:51.093810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:51.093816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:51.094288 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:51.094694 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:51.095173 (+ 479us) server_negotiation.cc:299] Negotiation successful
0807 03:00:51.095481 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:00:51.096695 25885 catalog_manager.cc:4841] TS df9bc41685b440de871cd6c501df0c40 (127.1.121.66:44999): tablet 9ff5bb40aaeb48de88cba490cd4438bd (table test-table [id=39e9271d57024ac6b8efe854dfcb540b]) successfully deleted
I20260807 03:00:51.096951 26165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:51.097121 26627 ts_tablet_manager.cc:1939] T 9ff5bb40aaeb48de88cba490cd4438bd P 9a469e5b203d459d8fb56fd4e52ddb8a: Deleting consensus metadata
I20260807 03:00:51.098904 25888 catalog_manager.cc:4841] TS 9a469e5b203d459d8fb56fd4e52ddb8a (127.1.121.67:34575): tablet 9ff5bb40aaeb48de88cba490cd4438bd (table test-table [id=39e9271d57024ac6b8efe854dfcb540b]) successfully deleted
I20260807 03:00:51.201333 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 25987
I20260807 03:00:51.229139 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 26179
Aug 07 03:00:51 dist-test-slave-3h5h krb5kdc[25838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 03:00:51.244406 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:51.230121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.1.121.66:44999)
0807 03:00:51.230316 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:51.230322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:51.230344 (+ 22us) server_negotiation.cc:407] Connection header received
0807 03:00:51.230458 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:51.230470 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:51.230534 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:51.230642 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:51.232021 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:51.232824 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:51.234107 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:51.234529 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:51.234678 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:51.239111 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:51.239123 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:51.239139 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:51.239204 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:51.241987 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:51.242659 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:51.242678 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:51.242680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:51.242804 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:51.243129 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:51.243132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:51.243134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:51.243503 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:51.243741 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:51.243979 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 03:00:51.244138 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:00:51.245342 26355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:51.340260 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 26179
I20260807 03:00:51.365139 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 26373
Aug 07 03:00:51 dist-test-slave-3h5h krb5kdc[25838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 03:00:51.381134 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:51.366020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.1.121.67:34575)
0807 03:00:51.366226 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:51.366234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:51.366260 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:00:51.366497 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:51.366502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:51.366570 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:51.366683 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:51.368055 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:51.368820 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:51.370182 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:51.370634 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:51.370779 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:51.375407 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:51.375425 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:51.375444 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:51.375484 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:51.378444 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:51.379283 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:51.379290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:51.379293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:51.379427 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:51.379789 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:51.379792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:51.379794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:51.380140 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:51.380369 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:51.380641 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 03:00:51.380812 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:00:51.381868 26553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:51.473158 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 26373
I20260807 03:00:51.498108 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 25856
I20260807 03:00:51.499357 25951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:51.611361 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 25856
2026-08-07T03:00:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4806 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T03:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:00:51Z Disabled control of system clock
I20260807 03:00:51.712934 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41109
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:41109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:00:52.069281 26646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:52.069598 26646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:52.069684 26646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:52.077195 26646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:00:52.077314 26646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:52.077363 26646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:52.077422 26646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:00:52.077467 26646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:00:52.086972 26646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:41109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:41109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:52.088497 26646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:52.090425 26646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:52.102885 26653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:52.102818 26655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:52.102823 26652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:52.103866 26646 server_base.cc:1034] running on GCE node
I20260807 03:00:52.104734 26646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:52.106308 26646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:52.107518 26646 hybrid_clock.cc:648] HybridClock initialized: now 1786071652107459 us; error 53 us; skew 500 ppm
I20260807 03:00:52.107897 26646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:00:52.114835 26646 webserver.cc:460] Webserver started at http://127.1.121.126:46709/ using document root <none> and password file <none>
I20260807 03:00:52.115614 26646 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:52.115744 26646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:52.116129 26646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:52.119302 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2"
format_stamp: "Formatted at 2026-08-07 03:00:52 on dist-test-slave-3h5h"
server_key: "8133d8b9b9cb44fedae12490e57e5123"
server_key_iv: "b8a685b69d228e58d2cc251f96e6e023"
server_key_version: "encryptionkey@0"
I20260807 03:00:52.120146 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2"
format_stamp: "Formatted at 2026-08-07 03:00:52 on dist-test-slave-3h5h"
server_key: "8133d8b9b9cb44fedae12490e57e5123"
server_key_iv: "b8a685b69d228e58d2cc251f96e6e023"
server_key_version: "encryptionkey@0"
I20260807 03:00:52.125785 26646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 03:00:52.129285 26666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:52.130625 26646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:00:52.130837 26646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2"
format_stamp: "Formatted at 2026-08-07 03:00:52 on dist-test-slave-3h5h"
server_key: "8133d8b9b9cb44fedae12490e57e5123"
server_key_iv: "b8a685b69d228e58d2cc251f96e6e023"
server_key_version: "encryptionkey@0"
I20260807 03:00:52.131076 26646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:52.143708 26646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:52.144583 26646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:52.144922 26646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:52.164988 26646 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:41109
I20260807 03:00:52.165009 26739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:41109 every 8 connection(s)
I20260807 03:00:52.166415 26646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:00:52.170300 26740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:52.174222 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 26646
I20260807 03:00:52.174413 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:00:52.174887 1509 external_mini_cluster.cc:1428] Setting key ab19f29393e16ed4f0cb0ebacf547b09
I20260807 03:00:52.181516 26740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2: Bootstrap starting.
I20260807 03:00:52.184589 26740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:52.185772 26740 log.cc:824] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:52.189391 26740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2: No bootstrap required, opened a new log
I20260807 03:00:52.189576 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:52.176913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.1.121.126:41109)
0807 03:00:52.177891 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:52.177902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:52.177944 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:00:52.180101 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:52.180125 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:52.180562 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:52.180994 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:52.181564 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:52.182951 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:52.184462 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:52.185303 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:52.185516 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:52.186840 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:52.186847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:52.186862 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:00:52.187535 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:52.187639 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:00:52.187665 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:52.187793 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:52.188001 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 03:00:52.188327 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":346,"threads_started":1}
I20260807 03:00:52.196806 26740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41109 } }
I20260807 03:00:52.197261 26740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:52.197427 26740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1fb0d14b694b33b2bb80c470e9a2d2, State: Initialized, Role: FOLLOWER
I20260807 03:00:52.198073 26740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41109 } }
I20260807 03:00:52.198381 26740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:00:52.198532 26740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:00:52.198736 26740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:52.201013 26740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41109 } }
I20260807 03:00:52.201529 26740 leader_election.cc:302] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b1fb0d14b694b33b2bb80c470e9a2d2; no voters:
I20260807 03:00:52.202132 26740 leader_election.cc:288] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:00:52.202286 26747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:00:52.203619 26747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b1fb0d14b694b33b2bb80c470e9a2d2, State: Running, Role: LEADER
I20260807 03:00:52.204210 26747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41109 } }
I20260807 03:00:52.204620 26740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:00:52.207448 26748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41109 } } }
I20260807 03:00:52.207911 26748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [sys.catalog]: This master's current role is: LEADER
I20260807 03:00:52.207419 26749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b1fb0d14b694b33b2bb80c470e9a2d2. Latest consensus state: current_term: 1 leader_uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1fb0d14b694b33b2bb80c470e9a2d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 41109 } } }
I20260807 03:00:52.208479 26749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2 [sys.catalog]: This master's current role is: LEADER
W20260807 03:00:52.214452 26767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:00:52.214564 26767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:00:52.214715 26757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:00:52.217146 26757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:00:52.226365 26757 catalog_manager.cc:1347] Generated new cluster ID: f36dac755eb64d05b7fd92f5b5204a4f
I20260807 03:00:52.226500 26757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:00:52.245223 26757 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:00:52.246543 26757 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:00:52.257118 26757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2: Generated new TSK 0
I20260807 03:00:52.258033 26757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:00:52.273442 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41109
--builtin_ntp_servers=127.1.121.84:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:00:52.637138 26772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:52.637480 26772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:52.637657 26772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:52.645069 26772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:52.645205 26772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:52.645377 26772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 03:00:52.654548 26772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:52.655992 26772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:52.657933 26772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:52.672271 26778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:52.672359 26779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:52.672950 26772 server_base.cc:1034] running on GCE node
W20260807 03:00:52.673003 26782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:52.673661 26772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:52.674681 26772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:52.675923 26772 hybrid_clock.cc:648] HybridClock initialized: now 1786071652675857 us; error 60 us; skew 500 ppm
I20260807 03:00:52.676265 26772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:00:52.678807 26772 webserver.cc:460] Webserver started at http://127.1.121.65:39769/ using document root <none> and password file <none>
I20260807 03:00:52.679575 26772 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:52.679718 26772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:52.680061 26772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:52.683076 26772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "22b6e024cf4e487caa277324d10091df"
format_stamp: "Formatted at 2026-08-07 03:00:52 on dist-test-slave-3h5h"
server_key: "fc30812c817f04967a688bbca85a8d2b"
server_key_iv: "8290073cd6991ffd867dce3409fa2bf6"
server_key_version: "encryptionkey@0"
I20260807 03:00:52.683874 26772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "22b6e024cf4e487caa277324d10091df"
format_stamp: "Formatted at 2026-08-07 03:00:52 on dist-test-slave-3h5h"
server_key: "fc30812c817f04967a688bbca85a8d2b"
server_key_iv: "8290073cd6991ffd867dce3409fa2bf6"
server_key_version: "encryptionkey@0"
I20260807 03:00:52.689338 26772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 03:00:52.694465 26793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:52.695849 26772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260807 03:00:52.696038 26772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "22b6e024cf4e487caa277324d10091df"
format_stamp: "Formatted at 2026-08-07 03:00:52 on dist-test-slave-3h5h"
server_key: "fc30812c817f04967a688bbca85a8d2b"
server_key_iv: "8290073cd6991ffd867dce3409fa2bf6"
server_key_version: "encryptionkey@0"
I20260807 03:00:52.696269 26772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:52.707762 26772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:52.708698 26772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:52.709080 26772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:52.710050 26772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:52.711659 26772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:52.711789 26772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:52.711921 26772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:52.712010 26772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:52.749547 26772 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:37689
I20260807 03:00:52.749605 26971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:37689 every 8 connection(s)
I20260807 03:00:52.750988 26772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 03:00:52.759317 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 26772
I20260807 03:00:52.759502 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 03:00:52.760043 1509 external_mini_cluster.cc:1428] Setting key d61aab06ab552ebc5042a1968270a701
I20260807 03:00:52.763481 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41109
--builtin_ntp_servers=127.1.121.84:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:00:52.766285 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:52.753792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:54937 (local address 127.1.121.126:41109)
0807 03:00:52.754282 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:52.754287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:52.755405 (+ 1118us) server_negotiation.cc:407] Connection header received
0807 03:00:52.757380 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:52.757386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:52.757483 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:52.757613 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:52.758792 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:52.759971 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:52.762604 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:52.763271 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:52.763422 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:52.765222 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:52.765231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:52.765240 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:00:52.765265 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:52.765326 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:00:52.765346 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:52.765446 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:52.765728 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 03:00:52.765843 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":266,"threads_started":1}
I20260807 03:00:52.767168 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:52.754258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41109 (local address 127.1.121.65:54937)
0807 03:00:52.755219 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:52.755266 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:52.757051 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:52.757868 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:52.757877 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:52.757976 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:52.758573 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:52.758590 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:52.760242 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:52.760247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:52.762435 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:52.762450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:52.763602 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:52.763611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:52.763919 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:52.764748 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:00:52.764751 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:52.764807 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:00:52.764968 (+ 161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:52.765483 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:52.765493 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:52.765500 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 03:00:52.765689 (+ 189us) client_negotiation.cc:240] Negotiation successful
0807 03:00:52.766005 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":279,"threads_started":1}
I20260807 03:00:52.768812 26972 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41109
I20260807 03:00:52.769178 26972 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:52.770149 26972 heartbeater.cc:511] Master 127.1.121.126:41109 requested a full tablet report, sending...
I20260807 03:00:52.772441 26692 ts_manager.cc:194] Registered new tserver with Master: 22b6e024cf4e487caa277324d10091df (127.1.121.65:37689)
I20260807 03:00:52.774011 26692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.65:54937
W20260807 03:00:53.114933 26976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:53.115222 26976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:53.115311 26976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:53.122699 26976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:53.122808 26976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:53.122916 26976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 03:00:53.132321 26976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:53.133602 26976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:53.135531 26976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:53.149689 26982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:53.149658 26986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:53.149637 26984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:53.150624 26976 server_base.cc:1034] running on GCE node
I20260807 03:00:53.151278 26976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:53.152267 26976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:53.153514 26976 hybrid_clock.cc:648] HybridClock initialized: now 1786071653153470 us; error 58 us; skew 500 ppm
I20260807 03:00:53.153908 26976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:00:53.156597 26976 webserver.cc:460] Webserver started at http://127.1.121.66:39625/ using document root <none> and password file <none>
I20260807 03:00:53.157450 26976 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:53.157640 26976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:53.157997 26976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:53.161228 26976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "5ae535ba6bab46d9b96fd5e9e33078c0"
format_stamp: "Formatted at 2026-08-07 03:00:53 on dist-test-slave-3h5h"
server_key: "76554c7fb77989a33ad98bc6d5ecc817"
server_key_iv: "84e204e8325cd99d30f5c838a8f03ab3"
server_key_version: "encryptionkey@0"
I20260807 03:00:53.162086 26976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "5ae535ba6bab46d9b96fd5e9e33078c0"
format_stamp: "Formatted at 2026-08-07 03:00:53 on dist-test-slave-3h5h"
server_key: "76554c7fb77989a33ad98bc6d5ecc817"
server_key_iv: "84e204e8325cd99d30f5c838a8f03ab3"
server_key_version: "encryptionkey@0"
I20260807 03:00:53.168532 26976 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 03:00:53.173058 26993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:53.174733 26976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:00:53.174901 26976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "5ae535ba6bab46d9b96fd5e9e33078c0"
format_stamp: "Formatted at 2026-08-07 03:00:53 on dist-test-slave-3h5h"
server_key: "76554c7fb77989a33ad98bc6d5ecc817"
server_key_iv: "84e204e8325cd99d30f5c838a8f03ab3"
server_key_version: "encryptionkey@0"
I20260807 03:00:53.175132 26976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:53.195503 26976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:53.196372 26976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:53.196702 26976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:53.197705 26976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:53.199137 26976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:53.199254 26976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:53.199407 26976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:53.199501 26976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:53.232326 26976 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:38173
I20260807 03:00:53.232400 27161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:38173 every 8 connection(s)
I20260807 03:00:53.233733 26976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 03:00:53.237907 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 26976
I20260807 03:00:53.238112 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 03:00:53.238720 1509 external_mini_cluster.cc:1428] Setting key 5c7f66559d53a38910f3a1ecffc6e23d
I20260807 03:00:53.241654 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41109
--builtin_ntp_servers=127.1.121.84:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:00:53.246855 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.236466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:33967 (local address 127.1.121.126:41109)
0807 03:00:53.236649 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:53.236656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:53.238362 (+ 1706us) server_negotiation.cc:407] Connection header received
0807 03:00:53.240410 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:53.240416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:53.240511 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:53.240627 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:53.241724 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.242520 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.244053 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.244455 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.244593 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.246039 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:53.246044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:53.246047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:00:53.246064 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:53.246095 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:00:53.246107 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:53.246208 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:53.246443 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 03:00:53.246540 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:00:53.247835 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.237086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41109 (local address 127.1.121.66:33967)
0807 03:00:53.238186 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:53.238243 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:53.240140 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:53.240907 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:53.240920 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:53.241004 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:53.241552 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.241568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.242685 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.242697 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.243896 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.243908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.244708 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.244720 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.245010 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.245716 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:00:53.245719 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:53.245756 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:00:53.245836 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:53.246219 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:53.246231 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:53.246246 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 03:00:53.246413 (+ 167us) client_negotiation.cc:240] Negotiation successful
0807 03:00:53.246801 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":287,"threads_started":1}
I20260807 03:00:53.249286 27162 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41109
I20260807 03:00:53.249578 27162 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:53.250272 27162 heartbeater.cc:511] Master 127.1.121.126:41109 requested a full tablet report, sending...
I20260807 03:00:53.251770 26692 ts_manager.cc:194] Registered new tserver with Master: 5ae535ba6bab46d9b96fd5e9e33078c0 (127.1.121.66:38173)
I20260807 03:00:53.253006 26692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.66:33967
W20260807 03:00:53.603130 27167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:53.603410 27167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:53.603590 27167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:53.611017 27167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:53.611119 27167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:53.611230 27167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 03:00:53.620283 27167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:41109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:53.621583 27167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:53.623385 27167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:53.637041 27174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:53.637286 27173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:53.637020 27176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:53.637851 27167 server_base.cc:1034] running on GCE node
I20260807 03:00:53.638430 27167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:53.639358 27167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:53.640573 27167 hybrid_clock.cc:648] HybridClock initialized: now 1786071653640524 us; error 35 us; skew 500 ppm
I20260807 03:00:53.640990 27167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:00:53.643355 27167 webserver.cc:460] Webserver started at http://127.1.121.67:35531/ using document root <none> and password file <none>
I20260807 03:00:53.644137 27167 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:53.644253 27167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:53.644586 27167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:53.647560 27167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "9e6275dc6428459989e4890963b9ea4a"
format_stamp: "Formatted at 2026-08-07 03:00:53 on dist-test-slave-3h5h"
server_key: "57bcdcfb8de45d77fea15584a00dd620"
server_key_iv: "a115570e560a616329df359db961f56d"
server_key_version: "encryptionkey@0"
I20260807 03:00:53.648310 27167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "9e6275dc6428459989e4890963b9ea4a"
format_stamp: "Formatted at 2026-08-07 03:00:53 on dist-test-slave-3h5h"
server_key: "57bcdcfb8de45d77fea15584a00dd620"
server_key_iv: "a115570e560a616329df359db961f56d"
server_key_version: "encryptionkey@0"
I20260807 03:00:53.653580 27167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 03:00:53.657389 27190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:53.658764 27167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:00:53.658941 27167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "9e6275dc6428459989e4890963b9ea4a"
format_stamp: "Formatted at 2026-08-07 03:00:53 on dist-test-slave-3h5h"
server_key: "57bcdcfb8de45d77fea15584a00dd620"
server_key_iv: "a115570e560a616329df359db961f56d"
server_key_version: "encryptionkey@0"
I20260807 03:00:53.659198 27167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:53.683849 27167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:53.684674 27167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:53.685048 27167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:53.686002 27167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:53.687433 27167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:53.687521 27167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:53.687595 27167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:53.687646 27167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:53.720932 27167 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:40207
I20260807 03:00:53.721025 27354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:40207 every 8 connection(s)
I20260807 03:00:53.722697 27167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 03:00:53.726052 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 27167
I20260807 03:00:53.726280 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 03:00:53.726897 1509 external_mini_cluster.cc:1428] Setting key 7d96f6d1a7ce775dd48b7fae8a27fc0a
I20260807 03:00:53.736321 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.725880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:34433 (local address 127.1.121.126:41109)
0807 03:00:53.726095 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:53.726102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:53.727622 (+ 1520us) server_negotiation.cc:407] Connection header received
0807 03:00:53.729933 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:53.729937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:53.730004 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:53.730085 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:53.731049 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.731767 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.733352 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.733784 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.733879 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.735412 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:53.735419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:53.735426 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:00:53.735450 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:53.735495 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:00:53.735513 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:53.735601 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:53.735887 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 03:00:53.735984 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 03:00:53.737388 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.726306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:41109 (local address 127.1.121.67:34433)
0807 03:00:53.727452 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:53.727490 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:53.729628 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:53.730257 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:53.730265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:53.730355 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:53.730911 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.730927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.731904 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.731908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.733227 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.733239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.733971 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.733976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.734161 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.734909 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:00:53.734915 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:53.734965 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:00:53.735108 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:53.735631 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:53.735642 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:53.735650 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 03:00:53.735872 (+ 222us) client_negotiation.cc:240] Negotiation successful
0807 03:00:53.736265 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":309,"threads_started":1}
I20260807 03:00:53.738682 27358 heartbeater.cc:347] Connected to a master server at 127.1.121.126:41109
I20260807 03:00:53.738991 27358 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:53.739799 27358 heartbeater.cc:511] Master 127.1.121.126:41109 requested a full tablet report, sending...
I20260807 03:00:53.741482 26692 ts_manager.cc:194] Registered new tserver with Master: 9e6275dc6428459989e4890963b9ea4a (127.1.121.67:40207)
I20260807 03:00:53.742374 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:00:53.742439 26692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.67:34433
I20260807 03:00:53.753909 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.747580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.1.121.126:41109)
0807 03:00:53.747878 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:53.747887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:53.748313 (+ 426us) server_negotiation.cc:407] Connection header received
0807 03:00:53.748524 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:53.748529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:53.748607 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:53.748722 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:53.749292 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.749977 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.751394 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.751816 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.751957 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.753177 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:53.753183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:53.753186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:00:53.753204 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:53.753236 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:00:53.753248 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:53.753357 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:53.753498 (+ 141us) server_negotiation.cc:299] Negotiation successful
0807 03:00:53.753591 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:00:53.757763 26692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:00:53.764628 26692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:00:53.777789 26972 heartbeater.cc:503] Master 127.1.121.126:41109 was elected leader, sending a full tablet report...
I20260807 03:00:53.801940 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.778396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.0.0.1:47130)
0807 03:00:53.792270 (+ 13874us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:53.792293 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:53.792445 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:53.793608 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:53.793618 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:53.793654 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:53.794205 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.794227 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.796191 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.796196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.798095 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.798120 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.800989 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.800994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.801202 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.801244 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 03:00:53.801379 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 03:00:53.801566 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13686,"thread_start_us":12187,"threads_started":1}
I20260807 03:00:53.801970 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.778663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.1.121.65:37689)
0807 03:00:53.791990 (+ 13327us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:53.792003 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:53.792425 (+ 422us) server_negotiation.cc:407] Connection header received
0807 03:00:53.792646 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:53.792658 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:53.793021 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:53.793276 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:53.794475 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.796022 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.798348 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.799863 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.799992 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.800884 (+ 892us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:53.801439 (+ 555us) server_negotiation.cc:299] Negotiation successful
0807 03:00:53.801611 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13079,"thread_start_us":13338,"threads_started":1}
I20260807 03:00:53.802825 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.778789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:40207 (local address 127.0.0.1:47834)
0807 03:00:53.794293 (+ 15504us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:53.794315 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:53.794438 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:53.795390 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:53.795395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:53.795421 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:53.795819 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.795831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.797825 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.797830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.799849 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.799865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.801965 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.801972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.802174 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.802209 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 03:00:53.802297 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 03:00:53.802428 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15357,"mutex_wait_us":100,"thread_start_us":15152,"threads_started":1}
I20260807 03:00:53.803293 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.777595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:38173 (local address 127.0.0.1:59604)
0807 03:00:53.790991 (+ 13396us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:53.791032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:53.791243 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:53.795055 (+ 3812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:53.795059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:53.795083 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:53.795505 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.795520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.797928 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.797931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.800295 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:53.800308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.802280 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.802290 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:53.802466 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.802495 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 03:00:53.802553 (+ 58us) client_negotiation.cc:240] Negotiation successful
0807 03:00:53.802674 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13148,"mutex_wait_us":83,"thread_start_us":14985,"threads_started":1}
I20260807 03:00:53.803797 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.777875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.1.121.66:38173)
0807 03:00:53.794598 (+ 16723us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:53.794605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:53.794630 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:00:53.794708 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:53.794713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:53.794869 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:53.795064 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:53.795758 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.797768 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.800577 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.802041 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.802936 (+ 895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.803218 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:53.803284 (+ 66us) server_negotiation.cc:299] Negotiation successful
0807 03:00:53.803446 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16551,"thread_start_us":16414,"threads_started":1}
I20260807 03:00:53.806208 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:53.780083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.1.121.67:40207)
0807 03:00:53.794075 (+ 13992us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:53.794086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:53.794574 (+ 488us) server_negotiation.cc:407] Connection header received
0807 03:00:53.794704 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:53.794712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:53.794936 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:53.795135 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:53.796047 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.797670 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.800083 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:53.801417 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:53.801551 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:53.801820 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:53.803202 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0807 03:00:53.805820 (+ 2618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13796,"thread_start_us":13593,"threads_started":1}
I20260807 03:00:53.806318 26874 tablet_service.cc:1467] Processing CreateTablet for tablet e11b89f6723243bd905da8f6393386fc (DEFAULT_TABLE table=test-table [id=a3faae6e83ad4a5396277c8c5f6ad9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:53.807834 26874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11b89f6723243bd905da8f6393386fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:53.808418 27059 tablet_service.cc:1467] Processing CreateTablet for tablet e11b89f6723243bd905da8f6393386fc (DEFAULT_TABLE table=test-table [id=a3faae6e83ad4a5396277c8c5f6ad9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:53.810067 27059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11b89f6723243bd905da8f6393386fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:53.809664 27260 tablet_service.cc:1467] Processing CreateTablet for tablet e11b89f6723243bd905da8f6393386fc (DEFAULT_TABLE table=test-table [id=a3faae6e83ad4a5396277c8c5f6ad9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:53.811017 27260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11b89f6723243bd905da8f6393386fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:53.820988 27379 tablet_bootstrap.cc:492] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df: Bootstrap starting.
I20260807 03:00:53.824378 27379 tablet_bootstrap.cc:654] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:53.825013 27380 tablet_bootstrap.cc:492] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: Bootstrap starting.
I20260807 03:00:53.825938 27379 log.cc:824] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:53.827549 27381 tablet_bootstrap.cc:492] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: Bootstrap starting.
I20260807 03:00:53.829326 27380 tablet_bootstrap.cc:654] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:53.830516 27381 tablet_bootstrap.cc:654] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:53.830546 27380 log.cc:824] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:53.831811 27379 tablet_bootstrap.cc:492] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df: No bootstrap required, opened a new log
I20260807 03:00:53.831779 27381 log.cc:824] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:53.832114 27379 ts_tablet_manager.cc:1397] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260807 03:00:53.834007 27380 tablet_bootstrap.cc:492] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: No bootstrap required, opened a new log
I20260807 03:00:53.834270 27380 ts_tablet_manager.cc:1397] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 03:00:53.835153 27381 tablet_bootstrap.cc:492] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: No bootstrap required, opened a new log
I20260807 03:00:53.835403 27381 ts_tablet_manager.cc:1397] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260807 03:00:53.842475 27381 raft_consensus.cc:348] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } }
I20260807 03:00:53.842872 27381 raft_consensus.cc:374] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:53.842983 27381 raft_consensus.cc:729] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6275dc6428459989e4890963b9ea4a, State: Initialized, Role: FOLLOWER
I20260807 03:00:53.842801 27379 raft_consensus.cc:348] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } }
I20260807 03:00:53.843231 27379 raft_consensus.cc:374] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:53.843334 27379 raft_consensus.cc:729] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b6e024cf4e487caa277324d10091df, State: Initialized, Role: FOLLOWER
I20260807 03:00:53.843725 27381 consensus_queue.cc:260] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } }
I20260807 03:00:53.843914 27379 consensus_queue.cc:260] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } }
I20260807 03:00:53.844954 27358 heartbeater.cc:503] Master 127.1.121.126:41109 was elected leader, sending a full tablet report...
I20260807 03:00:53.845621 27381 ts_tablet_manager.cc:1428] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260807 03:00:53.845250 27380 raft_consensus.cc:348] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } }
I20260807 03:00:53.846071 27380 raft_consensus.cc:374] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:53.846191 27380 raft_consensus.cc:729] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae535ba6bab46d9b96fd5e9e33078c0, State: Initialized, Role: FOLLOWER
I20260807 03:00:53.847028 27380 consensus_queue.cc:260] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } }
I20260807 03:00:53.848287 27162 heartbeater.cc:503] Master 127.1.121.126:41109 was elected leader, sending a full tablet report...
I20260807 03:00:53.849222 27380 ts_tablet_manager.cc:1428] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260807 03:00:53.850943 27379 ts_tablet_manager.cc:1428] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df: Time spent starting tablet: real 0.019s user 0.008s sys 0.012s
W20260807 03:00:53.974898 27359 tablet.cc:2369] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:00:53.985857 27163 tablet.cc:2369] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:00:54.003536 26973 tablet.cc:2369] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:00:54.007835 27393 raft_consensus.cc:482] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:00:54.008124 27393 raft_consensus.cc:504] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } }
I20260807 03:00:54.010002 27393 leader_election.cc:288] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22b6e024cf4e487caa277324d10091df (127.1.121.65:37689), 9e6275dc6428459989e4890963b9ea4a (127.1.121.67:40207)
I20260807 03:00:54.017271 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.010482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:42911 (local address 127.1.121.67:40207)
0807 03:00:54.010716 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:54.010725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:54.011402 (+ 677us) server_negotiation.cc:407] Connection header received
0807 03:00:54.011579 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:54.011585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:54.011684 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:54.011768 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:54.012443 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.013330 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.015543 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.016306 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.016523 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:54.016593 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:54.016928 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 03:00:54.017049 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:00:54.017292 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.010416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:39477 (local address 127.1.121.65:37689)
0807 03:00:54.010651 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:54.010661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:54.011581 (+ 920us) server_negotiation.cc:407] Connection header received
0807 03:00:54.011735 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:54.011760 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:54.011858 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:54.011965 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:54.012697 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.013490 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.015493 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.016288 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.016456 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:54.016543 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:54.016898 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 03:00:54.017029 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:00:54.017372 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.010396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.1.121.66:39477)
0807 03:00:54.011421 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.011440 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:54.011597 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:54.011989 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:54.011992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:54.012065 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:54.012515 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:54.012528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.013644 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.013648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:54.015294 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:54.015307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.016496 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.016504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:54.016689 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:54.016729 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 03:00:54.016802 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 03:00:54.016967 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":120,"spinlock_wait_cycles":24832,"thread_start_us":493,"threads_started":1}
I20260807 03:00:54.017652 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.010395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:40207 (local address 127.1.121.66:42911)
0807 03:00:54.011283 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.011299 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:54.011434 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:54.011817 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:54.011823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:54.011856 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:54.012284 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:54.012295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.013481 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.013487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:54.015305 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:54.015318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.016533 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.016539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:54.016735 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:54.016759 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 03:00:54.016899 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 03:00:54.017027 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":177,"thread_start_us":487,"threads_started":1}
I20260807 03:00:54.018404 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b89f6723243bd905da8f6393386fc" candidate_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b6e024cf4e487caa277324d10091df" is_pre_election: true
I20260807 03:00:54.018635 27291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b89f6723243bd905da8f6393386fc" candidate_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6275dc6428459989e4890963b9ea4a" is_pre_election: true
I20260807 03:00:54.018975 26906 raft_consensus.cc:2454] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ae535ba6bab46d9b96fd5e9e33078c0 in term 0.
I20260807 03:00:54.019160 27291 raft_consensus.cc:2454] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ae535ba6bab46d9b96fd5e9e33078c0 in term 0.
I20260807 03:00:54.019704 26995 leader_election.cc:302] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b6e024cf4e487caa277324d10091df, 5ae535ba6bab46d9b96fd5e9e33078c0; no voters:
I20260807 03:00:54.020131 27393 raft_consensus.cc:2781] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:00:54.020294 27393 raft_consensus.cc:482] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:00:54.020416 27393 raft_consensus.cc:3037] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:54.022769 27393 raft_consensus.cc:504] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } }
I20260807 03:00:54.023388 27393 leader_election.cc:288] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [CANDIDATE]: Term 1 election: Requested vote from peers 22b6e024cf4e487caa277324d10091df (127.1.121.65:37689), 9e6275dc6428459989e4890963b9ea4a (127.1.121.67:40207)
I20260807 03:00:54.023954 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b89f6723243bd905da8f6393386fc" candidate_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b6e024cf4e487caa277324d10091df"
I20260807 03:00:54.023953 27291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11b89f6723243bd905da8f6393386fc" candidate_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6275dc6428459989e4890963b9ea4a"
I20260807 03:00:54.024188 26906 raft_consensus.cc:3037] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:54.024176 27291 raft_consensus.cc:3037] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:54.027559 27291 raft_consensus.cc:2454] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ae535ba6bab46d9b96fd5e9e33078c0 in term 1.
I20260807 03:00:54.027559 26906 raft_consensus.cc:2454] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ae535ba6bab46d9b96fd5e9e33078c0 in term 1.
I20260807 03:00:54.028069 27000 leader_election.cc:302] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae535ba6bab46d9b96fd5e9e33078c0, 9e6275dc6428459989e4890963b9ea4a; no voters:
I20260807 03:00:54.028403 27393 raft_consensus.cc:2781] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:00:54.028937 27393 raft_consensus.cc:686] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5ae535ba6bab46d9b96fd5e9e33078c0, State: Running, Role: LEADER
I20260807 03:00:54.029675 27393 consensus_queue.cc:237] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } }
I20260807 03:00:54.034415 26692 catalog_manager.cc:5495] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ae535ba6bab46d9b96fd5e9e33078c0 (127.1.121.66). New cstate: current_term: 1 leader_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ae535ba6bab46d9b96fd5e9e33078c0" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 } health_report { overall_health: UNKNOWN } } }
I20260807 03:00:54.106549 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.100876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.1.121.66:38173)
0807 03:00:54.101099 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:54.101106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:54.101127 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:00:54.101256 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:54.101261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:54.101355 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:54.101523 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:00:54.101930 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.102626 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.104145 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.104630 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.104776 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:54.105328 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:00:54.105894 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:00:54.106078 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:54.106179 (+ 101us) server_negotiation.cc:299] Negotiation successful
0807 03:00:54.106247 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:00:54.115103 26906 raft_consensus.cc:1261] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [term 1 FOLLOWER]: Refusing update from remote peer 5ae535ba6bab46d9b96fd5e9e33078c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:00:54.115108 27291 raft_consensus.cc:1261] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [term 1 FOLLOWER]: Refusing update from remote peer 5ae535ba6bab46d9b96fd5e9e33078c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:00:54.116087 27398 consensus_queue.cc:1035] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6275dc6428459989e4890963b9ea4a" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:00:54.116446 27393 consensus_queue.cc:1035] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b6e024cf4e487caa277324d10091df" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:00:54.128273 27401 mvcc.cc:205] Tried to move back new op lower bound from 7315749495244644352 to 7315749494909190144. Current Snapshot: MvccSnapshot[applied={T|T < 7315749495244644352}]
I20260807 03:00:54.142747 26691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:00:54.143062 26691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:00:54.147153 26691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b1fb0d14b694b33b2bb80c470e9a2d2: Sending DeleteTablet for 3 replicas of tablet e11b89f6723243bd905da8f6393386fc
I20260807 03:00:54.148393 27059 tablet_service.cc:1514] Processing DeleteTablet for tablet e11b89f6723243bd905da8f6393386fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:54 UTC) from {username='slave'} at 127.0.0.1:59604
I20260807 03:00:54.148530 27260 tablet_service.cc:1514] Processing DeleteTablet for tablet e11b89f6723243bd905da8f6393386fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:54 UTC) from {username='slave'} at 127.0.0.1:47834
I20260807 03:00:54.148509 26874 tablet_service.cc:1514] Processing DeleteTablet for tablet e11b89f6723243bd905da8f6393386fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:54 UTC) from {username='slave'} at 127.0.0.1:47130
I20260807 03:00:54.152029 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 26772
I20260807 03:00:54.161614 27415 tablet_replica.cc:330] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: stopping tablet replica
I20260807 03:00:54.163064 27415 raft_consensus.cc:2229] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:00:54.165194 27415 raft_consensus.cc:2258] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:54.168735 27415 ts_tablet_manager.cc:1905] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:54.175689 27414 tablet_replica.cc:330] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: stopping tablet replica
I20260807 03:00:54.176254 27414 raft_consensus.cc:2229] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:00:54.176523 27418 tablet_replica.cc:330] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df: stopping tablet replica
I20260807 03:00:54.177030 27418 raft_consensus.cc:2229] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:00:54.177206 27414 raft_consensus.cc:2258] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:54.177678 27418 raft_consensus.cc:2258] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:54.180018 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.153360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.1.121.65:37689)
0807 03:00:54.153607 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:54.153616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:54.167246 (+ 13630us) server_negotiation.cc:407] Connection header received
0807 03:00:54.167833 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:54.167841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:54.167968 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:54.168055 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:54.170145 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.171378 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.173389 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.174064 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.174172 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:54.178113 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:54.178124 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:54.178143 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:00:54.178976 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:54.179040 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:00:54.179076 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:54.179189 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:54.179478 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 03:00:54.179607 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 03:00:54.180578 27418 ts_tablet_manager.cc:1905] T e11b89f6723243bd905da8f6393386fc P 22b6e024cf4e487caa277324d10091df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:54.181331 26965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:54.183070 27414 ts_tablet_manager.cc:1905] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:54.183904 27415 ts_tablet_manager.cc:1918] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:54.184178 27415 log.cc:1196] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/e11b89f6723243bd905da8f6393386fc
I20260807 03:00:54.184782 27415 ts_tablet_manager.cc:1939] T e11b89f6723243bd905da8f6393386fc P 9e6275dc6428459989e4890963b9ea4a: Deleting consensus metadata
I20260807 03:00:54.186866 26674 catalog_manager.cc:4841] TS 9e6275dc6428459989e4890963b9ea4a (127.1.121.67:40207): tablet e11b89f6723243bd905da8f6393386fc (table test-table [id=a3faae6e83ad4a5396277c8c5f6ad9a7]) successfully deleted
I20260807 03:00:54.190508 27414 ts_tablet_manager.cc:1918] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:54.190781 27414 log.cc:1196] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/e11b89f6723243bd905da8f6393386fc
I20260807 03:00:54.191488 27414 ts_tablet_manager.cc:1939] T e11b89f6723243bd905da8f6393386fc P 5ae535ba6bab46d9b96fd5e9e33078c0: Deleting consensus metadata
I20260807 03:00:54.193825 26671 catalog_manager.cc:4841] TS 5ae535ba6bab46d9b96fd5e9e33078c0 (127.1.121.66:38173): tablet e11b89f6723243bd905da8f6393386fc (table test-table [id=a3faae6e83ad4a5396277c8c5f6ad9a7]) successfully deleted
I20260807 03:00:54.280283 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 26772
I20260807 03:00:54.304351 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 26976
W20260807 03:00:54.304432 26667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.121.65:37689 (error 108)
I20260807 03:00:54.313239 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.306180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.0.0.1:47144)
0807 03:00:54.312863 (+ 6683us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.313020 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6490,"thread_start_us":6395,"threads_started":1}
W20260807 03:00:54.313558 26667 catalog_manager.cc:4584] TS 22b6e024cf4e487caa277324d10091df (127.1.121.65:37689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e11b89f6723243bd905da8f6393386fc: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37689: connect: Connection refused (error 111)
I20260807 03:00:54.314893 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.305836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.1.121.66:38173)
0807 03:00:54.306079 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:54.306085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:54.306111 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:00:54.306188 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:54.306193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:54.306319 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:54.306459 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:54.307008 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.308682 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.310746 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.311726 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.311894 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:54.313250 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:54.313259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:54.313274 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:00:54.314080 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:54.314128 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:00:54.314151 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:54.314282 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:54.314441 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 03:00:54.314544 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 03:00:54.315745 27156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:54.367311 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.366811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.0.0.1:47150)
0807 03:00:54.367042 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.367134 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 03:00:54.368371 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.367995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.0.0.1:47158)
0807 03:00:54.368170 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.368237 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 03:00:54.429090 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.428545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.0.0.1:47168)
0807 03:00:54.428733 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.428884 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 03:00:54.429080 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 26976
I20260807 03:00:54.434273 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.433678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.0.0.1:47174)
0807 03:00:54.433920 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.434030 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 03:00:54.454794 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 27167
I20260807 03:00:54.461716 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.455615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.1.121.67:40207)
0807 03:00:54.455822 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:54.455828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:54.455848 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:00:54.456000 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:54.456004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:54.456098 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:54.456191 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:54.456575 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.457394 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.458708 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:54.459112 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:54.459220 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:54.460244 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:54.460254 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:54.460271 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:00:54.460905 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:54.460974 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:00:54.461003 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:54.461125 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:54.461260 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 03:00:54.461350 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 03:00:54.462396 27349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:54.514942 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.514441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.0.0.1:47188)
0807 03:00:54.514668 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.514783 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260807 03:00:54.516119 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.515681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.0.0.1:47202)
0807 03:00:54.515905 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.515984 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260807 03:00:54.551774 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 27167
I20260807 03:00:54.576669 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 26646
I20260807 03:00:54.577903 26733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:54.689863 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:54.689351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37689 (local address 127.0.0.1:47214)
0807 03:00:54.689555 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:54.689693 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 03:00:54.689893 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 26646
2026-08-07T03:00:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3077 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:00:54 dist-test-slave-3h5h krb5kdc[27434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:00:54 dist-test-slave-3h5h krb5kdc[27434](info): set up 2 sockets
Aug 07 03:00:54 dist-test-slave-3h5h krb5kdc[27434](info): commencing operation
krb5kdc: starting...
W20260807 03:00:56.093811 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.002s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:00:56 dist-test-slave-3h5h krb5kdc[27434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:00:56.432996 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:43671
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:43671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:56.788923 27451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:56.789227 27451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:56.789330 27451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:56.796661 27451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:00:56.796778 27451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:56.796850 27451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:56.796911 27451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:00:56.796957 27451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:00:56.806891 27451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:43671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:56.808218 27451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:56.809571 27451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:56.821775 27458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:56.821820 27459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:56.821727 27461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:56.822736 27451 server_base.cc:1034] running on GCE node
I20260807 03:00:56.823508 27451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:56.824981 27451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:56.826208 27451 hybrid_clock.cc:648] HybridClock initialized: now 1786071656826139 us; error 48 us; skew 500 ppm
I20260807 03:00:56.826517 27451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:56 dist-test-slave-3h5h krb5kdc[27434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071656, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:56.829627 27451 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:00:56.831234 27451 webserver.cc:460] Webserver started at http://127.1.121.126:41723/ using document root <none> and password file <none>
I20260807 03:00:56.831888 27451 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:56.831991 27451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:56.832270 27451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:56.835351 27451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "df4815112170408d99ebd6169fba58cf"
format_stamp: "Formatted at 2026-08-07 03:00:56 on dist-test-slave-3h5h"
server_key: "e5829504e4acf77f295359fd2ca5d378"
server_key_iv: "899d6c0a0bccaeffd3671f624ed707ff"
server_key_version: "encryptionkey@0"
I20260807 03:00:56.836061 27451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "df4815112170408d99ebd6169fba58cf"
format_stamp: "Formatted at 2026-08-07 03:00:56 on dist-test-slave-3h5h"
server_key: "e5829504e4acf77f295359fd2ca5d378"
server_key_iv: "899d6c0a0bccaeffd3671f624ed707ff"
server_key_version: "encryptionkey@0"
I20260807 03:00:56.841347 27451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260807 03:00:56.844949 27472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:56.846275 27451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:00:56.846470 27451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "df4815112170408d99ebd6169fba58cf"
format_stamp: "Formatted at 2026-08-07 03:00:56 on dist-test-slave-3h5h"
server_key: "e5829504e4acf77f295359fd2ca5d378"
server_key_iv: "899d6c0a0bccaeffd3671f624ed707ff"
server_key_version: "encryptionkey@0"
I20260807 03:00:56.846666 27451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:56.878159 27451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:56.882361 27451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:56.882666 27451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:56.903012 27451 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:43671
I20260807 03:00:56.903014 27546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:43671 every 8 connection(s)
I20260807 03:00:56.904590 27451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:00:56.906507 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 27451
I20260807 03:00:56.906724 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:00:56.907287 1509 external_mini_cluster.cc:1428] Setting key cfa8bf2ece86dd55037973d7068ff952
I20260807 03:00:56.909570 27547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 03:00:56 dist-test-slave-3h5h krb5kdc[27434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:56.923744 27547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf: Bootstrap starting.
I20260807 03:00:56.927135 27547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:56.928212 27547 log.cc:824] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:56.931347 27547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf: No bootstrap required, opened a new log
I20260807 03:00:56.932088 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:56.909406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.1.121.126:43671)
0807 03:00:56.910362 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:56.910375 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:56.910420 (+ 45us) server_negotiation.cc:407] Connection header received
0807 03:00:56.912222 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:56.912245 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:56.912722 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:56.913163 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:56.915137 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:56.916515 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:56.918366 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:56.919030 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:56.919172 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:56.924232 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:56.924240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:56.924258 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:56.924295 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:56.927625 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:56.928380 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:56.928388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:56.928395 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:56.928571 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:56.929093 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:56.929097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:56.929099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:56.929658 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:56.930000 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:56.930504 (+ 504us) server_negotiation.cc:299] Negotiation successful
0807 03:00:56.930964 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":479,"threads_started":1}
I20260807 03:00:56.940390 27547 raft_consensus.cc:348] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4815112170408d99ebd6169fba58cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43671 } }
I20260807 03:00:56.940799 27547 raft_consensus.cc:374] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:56.940944 27547 raft_consensus.cc:729] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4815112170408d99ebd6169fba58cf, State: Initialized, Role: FOLLOWER
I20260807 03:00:56.941534 27547 consensus_queue.cc:260] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4815112170408d99ebd6169fba58cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43671 } }
I20260807 03:00:56.941753 27547 raft_consensus.cc:388] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:00:56.941854 27547 raft_consensus.cc:482] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:00:56.941987 27547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:56.944576 27547 raft_consensus.cc:504] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4815112170408d99ebd6169fba58cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43671 } }
I20260807 03:00:56.945109 27547 leader_election.cc:302] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df4815112170408d99ebd6169fba58cf; no voters:
I20260807 03:00:56.945665 27547 leader_election.cc:288] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:00:56.945822 27554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:00:56.946511 27554 raft_consensus.cc:686] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [term 1 LEADER]: Becoming Leader. State: Replica: df4815112170408d99ebd6169fba58cf, State: Running, Role: LEADER
I20260807 03:00:56.947068 27554 consensus_queue.cc:237] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4815112170408d99ebd6169fba58cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43671 } }
I20260807 03:00:56.947856 27547 sys_catalog.cc:564] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:00:56.950384 27556 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader df4815112170408d99ebd6169fba58cf. Latest consensus state: current_term: 1 leader_uuid: "df4815112170408d99ebd6169fba58cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4815112170408d99ebd6169fba58cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43671 } } }
I20260807 03:00:56.950930 27556 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [sys.catalog]: This master's current role is: LEADER
I20260807 03:00:56.950624 27555 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df4815112170408d99ebd6169fba58cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4815112170408d99ebd6169fba58cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 43671 } } }
I20260807 03:00:56.951712 27555 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf [sys.catalog]: This master's current role is: LEADER
I20260807 03:00:56.957633 27563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:00:56.962877 27563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:00:56.971946 27563 catalog_manager.cc:1347] Generated new cluster ID: 44a9d020c7da47638e6e0af3788850e9
I20260807 03:00:56.972064 27563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:00:56.995594 27563 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:00:56.997056 27563 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:00:57.014504 27563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf: Generated new TSK 0
I20260807 03:00:57.015659 27563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 03:00:57.100451 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:43671
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:57.455621 27583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:57.455904 27583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:57.455998 27583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:57.463552 27583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:57.463665 27583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:57.463778 27583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 03:00:57.473534 27583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:43671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:57.474906 27583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:57.476239 27583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:57.489401 27593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:57.489384 27592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:57.489467 27595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:57.490244 27583 server_base.cc:1034] running on GCE node
I20260807 03:00:57.490834 27583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:57.491696 27583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:57.492933 27583 hybrid_clock.cc:648] HybridClock initialized: now 1786071657492854 us; error 61 us; skew 500 ppm
I20260807 03:00:57.493203 27583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:57 dist-test-slave-3h5h krb5kdc[27434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071657, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:57.501314 27583 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 03:00:57.503003 27583 webserver.cc:460] Webserver started at http://127.1.121.65:34551/ using document root <none> and password file <none>
I20260807 03:00:57.503758 27583 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:57.503901 27583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:57.504248 27583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:57.507365 27583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4"
format_stamp: "Formatted at 2026-08-07 03:00:57 on dist-test-slave-3h5h"
server_key: "09dbc6970830045379b77dff699819b2"
server_key_iv: "328041ec0c4e0fc6cbba0fa4b9e1b760"
server_key_version: "encryptionkey@0"
I20260807 03:00:57.508172 27583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4"
format_stamp: "Formatted at 2026-08-07 03:00:57 on dist-test-slave-3h5h"
server_key: "09dbc6970830045379b77dff699819b2"
server_key_iv: "328041ec0c4e0fc6cbba0fa4b9e1b760"
server_key_version: "encryptionkey@0"
I20260807 03:00:57.513631 27583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:00:57.521835 27604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:57.523437 27583 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260807 03:00:57.523591 27583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4"
format_stamp: "Formatted at 2026-08-07 03:00:57 on dist-test-slave-3h5h"
server_key: "09dbc6970830045379b77dff699819b2"
server_key_iv: "328041ec0c4e0fc6cbba0fa4b9e1b760"
server_key_version: "encryptionkey@0"
I20260807 03:00:57.523787 27583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:57.549253 27583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:57.553329 27583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:57.553581 27583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:57.554519 27583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:57.555827 27583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:57.555899 27583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:57.555984 27583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:57.556030 27583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:57.590374 27583 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:37887
I20260807 03:00:57.590469 27765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:37887 every 8 connection(s)
I20260807 03:00:57.591713 27583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 03:00:57.594061 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 27583
I20260807 03:00:57.594255 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 03:00:57.594861 1509 external_mini_cluster.cc:1428] Setting key 23f1ecbd221a2e79539d57d543b23398
Aug 07 03:00:57 dist-test-slave-3h5h krb5kdc[27434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071657, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:57.613756 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:57.594357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:34405 (local address 127.1.121.126:43671)
0807 03:00:57.594904 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:57.594912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:57.596467 (+ 1555us) server_negotiation.cc:407] Connection header received
0807 03:00:57.598107 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:57.598116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:57.598225 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:57.598349 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:57.599773 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:57.600939 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:57.602428 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:57.603124 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:57.603284 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:57.606917 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:57.606925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:57.606932 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:57.606978 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:57.610631 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:57.611357 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:57.611364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:57.611367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:57.611503 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:57.611938 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:57.611946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:57.611949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:57.612242 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:57.612405 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:57.613289 (+ 884us) server_negotiation.cc:299] Negotiation successful
0807 03:00:57.613474 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":264,"threads_started":1}
I20260807 03:00:57.615306 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:57.595032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43671 (local address 127.1.121.65:34405)
0807 03:00:57.595872 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:57.595918 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:57.597839 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:57.598549 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:57.598557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:57.599058 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:57.599587 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:57.599603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:57.601090 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:57.601094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:57.602277 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:57.602294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:57.603341 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:57.603346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:57.603533 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:57.604228 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:00:57.604250 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:57.606672 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:57.610777 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:57.610784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:57.610798 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:57.611227 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:57.611619 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:57.611624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:57.611628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:57.611818 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:57.612376 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:57.612384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:57.612859 (+ 475us) client_negotiation.cc:769] Sending connection context
0807 03:00:57.613195 (+ 336us) client_negotiation.cc:240] Negotiation successful
0807 03:00:57.613797 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":344,"threads_started":1}
I20260807 03:00:57.617062 27768 heartbeater.cc:347] Connected to a master server at 127.1.121.126:43671
I20260807 03:00:57.617408 27768 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:57.618341 27768 heartbeater.cc:511] Master 127.1.121.126:43671 requested a full tablet report, sending...
I20260807 03:00:57.620579 27503 ts_manager.cc:194] Registered new tserver with Master: 3fd1c36a0a9e4fd988a6493dd8e57fc4 (127.1.121.65:37887)
I20260807 03:00:57.622113 27503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:34405
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 03:00:57.673601 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:43671
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:58.031090 27776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:58.031373 27776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:58.031571 27776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:58.038923 27776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:58.039041 27776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:58.039247 27776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 03:00:58.049288 27776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:43671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:58.050668 27776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:58.052030 27776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:00:58.066738 27781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:58.066687 27782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:58.066645 27786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:58.067725 27776 server_base.cc:1034] running on GCE node
I20260807 03:00:58.068373 27776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:58.069458 27776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:58.070695 27776 hybrid_clock.cc:648] HybridClock initialized: now 1786071658070640 us; error 47 us; skew 500 ppm
I20260807 03:00:58.071028 27776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:58 dist-test-slave-3h5h krb5kdc[27434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071658, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:58.074347 27776 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 03:00:58.075882 27776 webserver.cc:460] Webserver started at http://127.1.121.66:37565/ using document root <none> and password file <none>
I20260807 03:00:58.076607 27776 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:58.076767 27776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:58.077131 27776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:58.080170 27776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "a78b175225d64d6490920fe94414dca7"
format_stamp: "Formatted at 2026-08-07 03:00:58 on dist-test-slave-3h5h"
server_key: "56312853a1bcd756015de822a9f7f401"
server_key_iv: "54ff486ca9f68f84316f6c12dc905e3f"
server_key_version: "encryptionkey@0"
I20260807 03:00:58.081045 27776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "a78b175225d64d6490920fe94414dca7"
format_stamp: "Formatted at 2026-08-07 03:00:58 on dist-test-slave-3h5h"
server_key: "56312853a1bcd756015de822a9f7f401"
server_key_iv: "54ff486ca9f68f84316f6c12dc905e3f"
server_key_version: "encryptionkey@0"
I20260807 03:00:58.086321 27776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:00:58.089905 27797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:58.091184 27776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:00:58.091370 27776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "a78b175225d64d6490920fe94414dca7"
format_stamp: "Formatted at 2026-08-07 03:00:58 on dist-test-slave-3h5h"
server_key: "56312853a1bcd756015de822a9f7f401"
server_key_iv: "54ff486ca9f68f84316f6c12dc905e3f"
server_key_version: "encryptionkey@0"
I20260807 03:00:58.091581 27776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:58.115617 27776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:58.119889 27776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:58.120208 27776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:58.121255 27776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:58.122609 27776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:58.122720 27776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:58.122870 27776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:58.122946 27776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:58.158461 27776 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:33127
I20260807 03:00:58.158579 27965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:33127 every 8 connection(s)
I20260807 03:00:58.160030 27776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 03:00:58.167328 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 27776
I20260807 03:00:58.167531 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 03:00:58.168081 1509 external_mini_cluster.cc:1428] Setting key 7c1b02798b96fd7c2b77c20883ddde2b
Aug 07 03:00:58 dist-test-slave-3h5h krb5kdc[27434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071658, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:58.182518 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.162787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:51573 (local address 127.1.121.126:43671)
0807 03:00:58.163283 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:58.163288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:58.164122 (+ 834us) server_negotiation.cc:407] Connection header received
0807 03:00:58.166133 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:58.166140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:58.166253 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:58.166377 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:58.168450 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.169391 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.171645 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.172071 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.172174 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.176461 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:58.176466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:58.176470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:58.176504 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:58.179310 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:58.180078 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:58.180088 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:58.180090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:58.180219 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:58.180662 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:58.180673 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:58.180675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:58.181008 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:58.181182 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:58.181890 (+ 708us) server_negotiation.cc:299] Negotiation successful
0807 03:00:58.182120 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":264,"threads_started":1}
I20260807 03:00:58.183604 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.163225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43671 (local address 127.1.121.66:51573)
0807 03:00:58.163950 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:58.163996 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:58.165848 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:58.166621 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:58.166630 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:58.167372 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:58.168234 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.168262 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.169559 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.169564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.171475 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.171494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.172309 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.172319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.172658 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.173450 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:00:58.173487 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:58.176220 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:58.179460 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:58.179469 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:58.179489 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:58.179940 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:58.180338 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:58.180343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:58.180347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:58.180538 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:58.181154 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:58.181162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:58.181525 (+ 363us) client_negotiation.cc:769] Sending connection context
0807 03:00:58.181799 (+ 274us) client_negotiation.cc:240] Negotiation successful
0807 03:00:58.182377 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":279,"threads_started":1}
I20260807 03:00:58.185124 27966 heartbeater.cc:347] Connected to a master server at 127.1.121.126:43671
I20260807 03:00:58.185513 27966 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:58.186501 27966 heartbeater.cc:511] Master 127.1.121.126:43671 requested a full tablet report, sending...
I20260807 03:00:58.188212 27503 ts_manager.cc:194] Registered new tserver with Master: a78b175225d64d6490920fe94414dca7 (127.1.121.66:33127)
I20260807 03:00:58.189088 27503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:51573
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 03:00:58.243664 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:43671
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:00:58.599359 27975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:00:58.599609 27975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:00:58.599714 27975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:00:58.607088 27975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:00:58.607204 27975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:00:58.607318 27975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 03:00:58.617467 27975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:43671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:00:58.618700 27975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:00:58.620013 27975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 03:00:58.625824 27768 heartbeater.cc:503] Master 127.1.121.126:43671 was elected leader, sending a full tablet report...
W20260807 03:00:58.634447 27980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:58.634536 27987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:00:58.634559 27983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:00:58.635344 27975 server_base.cc:1034] running on GCE node
I20260807 03:00:58.636044 27975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:00:58.637041 27975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:00:58.638243 27975 hybrid_clock.cc:648] HybridClock initialized: now 1786071658638197 us; error 44 us; skew 500 ppm
I20260807 03:00:58.638559 27975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:00:58 dist-test-slave-3h5h krb5kdc[27434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071658, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:00:58.641700 27975 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 03:00:58.643108 27975 webserver.cc:460] Webserver started at http://127.1.121.67:44291/ using document root <none> and password file <none>
I20260807 03:00:58.643795 27975 fs_manager.cc:359] Metadata directory not provided
I20260807 03:00:58.643929 27975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:00:58.644261 27975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:00:58.647212 27975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "fc252cb4508a4815afa3b476af676628"
format_stamp: "Formatted at 2026-08-07 03:00:58 on dist-test-slave-3h5h"
server_key: "32716c483a65f3f91508449cdda5bed8"
server_key_iv: "0068e37c1a5e1d8d2d188f1dc12a1b09"
server_key_version: "encryptionkey@0"
I20260807 03:00:58.648147 27975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "fc252cb4508a4815afa3b476af676628"
format_stamp: "Formatted at 2026-08-07 03:00:58 on dist-test-slave-3h5h"
server_key: "32716c483a65f3f91508449cdda5bed8"
server_key_iv: "0068e37c1a5e1d8d2d188f1dc12a1b09"
server_key_version: "encryptionkey@0"
I20260807 03:00:58.653611 27975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 03:00:58.657171 27996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:58.658634 27975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:00:58.658818 27975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "fc252cb4508a4815afa3b476af676628"
format_stamp: "Formatted at 2026-08-07 03:00:58 on dist-test-slave-3h5h"
server_key: "32716c483a65f3f91508449cdda5bed8"
server_key_iv: "0068e37c1a5e1d8d2d188f1dc12a1b09"
server_key_version: "encryptionkey@0"
I20260807 03:00:58.659013 27975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:00:58.675226 27975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:00:58.679222 27975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:00:58.679517 27975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:00:58.680505 27975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:00:58.681918 27975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:00:58.681993 27975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:58.682156 27975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:00:58.682209 27975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:00:58.716099 27975 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:37803
I20260807 03:00:58.716209 28160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:37803 every 8 connection(s)
I20260807 03:00:58.717518 27975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 03:00:58.727357 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 27975
I20260807 03:00:58.727555 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 03:00:58.728080 1509 external_mini_cluster.cc:1428] Setting key 185b4662104fd9d33f226eb6f78f94f2
Aug 07 03:00:58 dist-test-slave-3h5h krb5kdc[27434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071658, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:00:58.738183 28164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.720225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:56037 (local address 127.1.121.126:43671)
0807 03:00:58.720640 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:58.720645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:58.721698 (+ 1053us) server_negotiation.cc:407] Connection header received
0807 03:00:58.723484 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:58.723488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:58.723572 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:58.723672 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:58.725195 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.725945 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.727806 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.728476 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.728609 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.732624 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:58.732628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:58.732632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:58.732665 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:58.735384 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:58.736081 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:58.736085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:58.736087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:58.736214 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:58.736612 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:58.736615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:58.736617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:58.736958 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:58.737082 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:58.737733 (+ 651us) server_negotiation.cc:299] Negotiation successful
0807 03:00:58.737902 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":199,"threads_started":1}
I20260807 03:00:58.739476 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.720768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:43671 (local address 127.1.121.67:56037)
0807 03:00:58.721527 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:58.721573 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:58.723227 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:58.723873 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:58.723881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:58.724397 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:00:58.725026 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.725047 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.726079 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.726082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.727656 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.727673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.728701 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.728706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.728962 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.729745 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:00:58.729767 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:00:58.732431 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:00:58.735508 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:58.735515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:58.735528 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:58.735969 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:58.736307 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:00:58.736310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:00:58.736312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:00:58.736493 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:00:58.737077 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:00:58.737083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:00:58.737397 (+ 314us) client_negotiation.cc:769] Sending connection context
0807 03:00:58.737626 (+ 229us) client_negotiation.cc:240] Negotiation successful
0807 03:00:58.738016 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":287,"threads_started":1}
I20260807 03:00:58.741240 28161 heartbeater.cc:347] Connected to a master server at 127.1.121.126:43671
I20260807 03:00:58.741612 28161 heartbeater.cc:464] Registering TS with master...
I20260807 03:00:58.742470 28161 heartbeater.cc:511] Master 127.1.121.126:43671 requested a full tablet report, sending...
I20260807 03:00:58.744172 27501 ts_manager.cc:194] Registered new tserver with Master: fc252cb4508a4815afa3b476af676628 (127.1.121.67:37803)
I20260807 03:00:58.745437 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:00:58.745553 27501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:56037
I20260807 03:00:58.763502 28164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.750188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.1.121.126:43671)
0807 03:00:58.750328 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:58.750332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:58.750811 (+ 479us) server_negotiation.cc:407] Connection header received
0807 03:00:58.750968 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:58.750971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:58.751034 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:58.751115 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:58.752611 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.753396 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.754711 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.755108 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.755211 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.758249 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:58.758254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:58.758257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:58.758288 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:58.760896 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:58.761617 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:58.761621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:58.761623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:58.761738 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:58.762102 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:58.762105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:58.762107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:58.762408 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:58.762603 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:58.762985 (+ 382us) server_negotiation.cc:299] Negotiation successful
0807 03:00:58.763200 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260807 03:00:58.767627 27501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:00:58.774528 27501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:00:58.810900 28175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.787163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:33127 (local address 127.0.0.1:36408)
0807 03:00:58.802046 (+ 14883us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:58.802064 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:58.802204 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:58.802977 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:58.802988 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:58.803028 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:58.803771 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.803812 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.805628 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.805637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.808231 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.808247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.809735 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.809742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.809930 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.809988 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 03:00:58.810179 (+ 191us) client_negotiation.cc:240] Negotiation successful
0807 03:00:58.810378 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14755,"mutex_wait_us":2,"thread_start_us":12650,"threads_started":1}
I20260807 03:00:58.811040 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.787254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.1.121.66:33127)
0807 03:00:58.801196 (+ 13942us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:58.801209 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:58.802351 (+ 1142us) server_negotiation.cc:407] Connection header received
0807 03:00:58.802478 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:58.802488 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:58.802716 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:58.803223 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:58.804009 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.805495 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.808433 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.809528 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.809693 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.809919 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:58.810489 (+ 570us) server_negotiation.cc:299] Negotiation successful
0807 03:00:58.810690 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13731,"thread_start_us":15490,"threads_started":1}
I20260807 03:00:58.811038 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.786860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:37803 (local address 127.0.0.1:52230)
0807 03:00:58.800247 (+ 13387us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:58.800298 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:58.800473 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:58.803036 (+ 2563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:58.803044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:58.803082 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:58.804065 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.804081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.806036 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.806041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.808321 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.808339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.810193 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.810199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.810366 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.810395 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 03:00:58.810488 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 03:00:58.810627 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13109,"mutex_wait_us":37,"thread_start_us":15286,"threads_started":1}
I20260807 03:00:58.811726 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.787125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.1.121.67:37803)
0807 03:00:58.801791 (+ 14666us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:58.801802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:58.801842 (+ 40us) server_negotiation.cc:407] Connection header received
0807 03:00:58.801961 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:58.801972 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:58.802269 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:58.802490 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:58.804279 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.805879 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.808486 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.809962 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.810124 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.810366 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:58.811168 (+ 802us) server_negotiation.cc:299] Negotiation successful
0807 03:00:58.811332 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14452,"thread_start_us":14252,"threads_started":1}
I20260807 03:00:58.812028 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.790876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.1.121.65:37887)
0807 03:00:58.802892 (+ 12016us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:58.802903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:58.803709 (+ 806us) server_negotiation.cc:407] Connection header received
0807 03:00:58.803908 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:58.803918 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:58.804215 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:58.804413 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:58.805021 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.806501 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.808623 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.810032 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.810155 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.810365 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:58.811142 (+ 777us) server_negotiation.cc:299] Negotiation successful
0807 03:00:58.811335 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11811,"thread_start_us":11622,"threads_started":1}
I20260807 03:00:58.812119 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.789978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37887 (local address 127.0.0.1:40140)
0807 03:00:58.803554 (+ 13576us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:58.803579 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:58.803725 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:58.804416 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:58.804424 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:58.804455 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:58.804820 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.804872 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.806674 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.806683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.808449 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.808465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.810543 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.810549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.810724 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.810764 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 03:00:58.810865 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 03:00:58.810995 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13358,"mutex_wait_us":1,"thread_start_us":13087,"threads_started":1}
I20260807 03:00:58.815186 27863 tablet_service.cc:1467] Processing CreateTablet for tablet 5a44626af202438b92c6a651ce2a14d2 (DEFAULT_TABLE table=test-table [id=827c17502ecd4157994d5450b997fb57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:58.815284 27671 tablet_service.cc:1467] Processing CreateTablet for tablet 5a44626af202438b92c6a651ce2a14d2 (DEFAULT_TABLE table=test-table [id=827c17502ecd4157994d5450b997fb57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:58.816481 27863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44626af202438b92c6a651ce2a14d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:58.816633 27671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44626af202438b92c6a651ce2a14d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:58.818548 28071 tablet_service.cc:1467] Processing CreateTablet for tablet 5a44626af202438b92c6a651ce2a14d2 (DEFAULT_TABLE table=test-table [id=827c17502ecd4157994d5450b997fb57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:00:58.820031 28071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44626af202438b92c6a651ce2a14d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:00:58.831243 28182 tablet_bootstrap.cc:492] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: Bootstrap starting.
I20260807 03:00:58.832545 28183 tablet_bootstrap.cc:492] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: Bootstrap starting.
I20260807 03:00:58.835124 28185 tablet_bootstrap.cc:492] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: Bootstrap starting.
I20260807 03:00:58.835407 28182 tablet_bootstrap.cc:654] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:58.836180 28183 tablet_bootstrap.cc:654] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:58.836861 28182 log.cc:824] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:58.837491 28183 log.cc:824] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:58.838421 28185 tablet_bootstrap.cc:654] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: Neither blocks nor log segments found. Creating new log.
I20260807 03:00:58.839598 28185 log.cc:824] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: Log is configured to *not* fsync() on all Append() calls
I20260807 03:00:58.840014 28182 tablet_bootstrap.cc:492] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: No bootstrap required, opened a new log
I20260807 03:00:58.840310 28182 ts_tablet_manager.cc:1397] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260807 03:00:58.840876 28183 tablet_bootstrap.cc:492] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: No bootstrap required, opened a new log
I20260807 03:00:58.841202 28183 ts_tablet_manager.cc:1397] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260807 03:00:58.844288 28185 tablet_bootstrap.cc:492] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: No bootstrap required, opened a new log
I20260807 03:00:58.844564 28185 ts_tablet_manager.cc:1397] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260807 03:00:58.848793 28182 raft_consensus.cc:348] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } }
I20260807 03:00:58.849284 28182 raft_consensus.cc:374] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:58.849388 28182 raft_consensus.cc:729] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78b175225d64d6490920fe94414dca7, State: Initialized, Role: FOLLOWER
I20260807 03:00:58.850080 28182 consensus_queue.cc:260] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } }
I20260807 03:00:58.851262 27966 heartbeater.cc:503] Master 127.1.121.126:43671 was elected leader, sending a full tablet report...
I20260807 03:00:58.851922 28182 ts_tablet_manager.cc:1428] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260807 03:00:58.852037 28183 raft_consensus.cc:348] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } }
I20260807 03:00:58.852456 28183 raft_consensus.cc:374] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:58.852547 28183 raft_consensus.cc:729] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd1c36a0a9e4fd988a6493dd8e57fc4, State: Initialized, Role: FOLLOWER
I20260807 03:00:58.853226 28183 consensus_queue.cc:260] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } }
I20260807 03:00:58.855110 28183 ts_tablet_manager.cc:1428] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260807 03:00:58.855628 28185 raft_consensus.cc:348] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } }
I20260807 03:00:58.856093 28185 raft_consensus.cc:374] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:00:58.856247 28185 raft_consensus.cc:729] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc252cb4508a4815afa3b476af676628, State: Initialized, Role: FOLLOWER
I20260807 03:00:58.857014 28185 consensus_queue.cc:260] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } }
I20260807 03:00:58.858224 28161 heartbeater.cc:503] Master 127.1.121.126:43671 was elected leader, sending a full tablet report...
I20260807 03:00:58.859923 28185 ts_tablet_manager.cc:1428] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260807 03:00:58.912442 27967 tablet.cc:2369] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:00:58.921833 28192 raft_consensus.cc:482] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:00:58.922168 28192 raft_consensus.cc:504] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } }
I20260807 03:00:58.923904 28192 leader_election.cc:288] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc252cb4508a4815afa3b476af676628 (127.1.121.67:37803), a78b175225d64d6490920fe94414dca7 (127.1.121.66:33127)
I20260807 03:00:58.933575 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.924390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:54877 (local address 127.1.121.66:33127)
0807 03:00:58.924596 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:58.924602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:58.925525 (+ 923us) server_negotiation.cc:407] Connection header received
0807 03:00:58.925719 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:58.925725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:58.925807 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:58.925906 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:58.926739 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.927920 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.931159 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.932383 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.932553 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.932630 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:58.933178 (+ 548us) server_negotiation.cc:299] Negotiation successful
0807 03:00:58.933311 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:00:58.933574 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.924396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:35655 (local address 127.1.121.67:37803)
0807 03:00:58.924595 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:58.924603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:58.925466 (+ 863us) server_negotiation.cc:407] Connection header received
0807 03:00:58.925719 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:58.925725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:58.925807 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:58.925907 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:00:58.926837 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.927959 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.931149 (+ 3190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.932399 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.932556 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.932630 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:58.933174 (+ 544us) server_negotiation.cc:299] Negotiation successful
0807 03:00:58.933311 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:00:58.933826 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.924224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:33127 (local address 127.1.121.65:54877)
0807 03:00:58.925333 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:58.925354 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:58.925562 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:58.925941 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:58.925946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:58.925995 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:58.926569 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.926587 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.928078 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.928083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.930956 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.930973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.932694 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.932700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.932971 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.933023 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 03:00:58.933125 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 03:00:58.933267 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":342,"spinlock_wait_cycles":49664,"thread_start_us":663,"threads_started":1}
I20260807 03:00:58.933851 28194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.924296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:37803 (local address 127.1.121.65:35655)
0807 03:00:58.925333 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0807 03:00:58.925354 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:00:58.925516 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:00:58.925941 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:00:58.925946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:00:58.925994 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:00:58.926646 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.926659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.928086 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.928089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.930957 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:00:58.930981 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.932688 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.932695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:00:58.932965 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.933023 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 03:00:58.933126 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 03:00:58.933267 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":267,"spinlock_wait_cycles":13568,"thread_start_us":602,"threads_started":1}
I20260807 03:00:58.935115 28099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44626af202438b92c6a651ce2a14d2" candidate_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc252cb4508a4815afa3b476af676628" is_pre_election: true
I20260807 03:00:58.935204 27896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44626af202438b92c6a651ce2a14d2" candidate_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78b175225d64d6490920fe94414dca7" is_pre_election: true
I20260807 03:00:58.935783 27896 raft_consensus.cc:2454] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fd1c36a0a9e4fd988a6493dd8e57fc4 in term 0.
I20260807 03:00:58.935783 28099 raft_consensus.cc:2454] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fd1c36a0a9e4fd988a6493dd8e57fc4 in term 0.
I20260807 03:00:58.936527 27609 leader_election.cc:302] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd1c36a0a9e4fd988a6493dd8e57fc4, a78b175225d64d6490920fe94414dca7; no voters:
I20260807 03:00:58.936944 28192 raft_consensus.cc:2781] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:00:58.937111 28192 raft_consensus.cc:482] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:00:58.937250 28192 raft_consensus.cc:3037] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:58.939524 28192 raft_consensus.cc:504] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } }
I20260807 03:00:58.940263 28192 leader_election.cc:288] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [CANDIDATE]: Term 1 election: Requested vote from peers fc252cb4508a4815afa3b476af676628 (127.1.121.67:37803), a78b175225d64d6490920fe94414dca7 (127.1.121.66:33127)
I20260807 03:00:58.940789 28099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44626af202438b92c6a651ce2a14d2" candidate_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc252cb4508a4815afa3b476af676628"
I20260807 03:00:58.940811 27896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44626af202438b92c6a651ce2a14d2" candidate_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78b175225d64d6490920fe94414dca7"
I20260807 03:00:58.941080 28099 raft_consensus.cc:3037] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:58.941110 27896 raft_consensus.cc:3037] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:00:58.943444 27896 raft_consensus.cc:2454] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fd1c36a0a9e4fd988a6493dd8e57fc4 in term 1.
I20260807 03:00:58.943606 28099 raft_consensus.cc:2454] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fd1c36a0a9e4fd988a6493dd8e57fc4 in term 1.
I20260807 03:00:58.944010 27609 leader_election.cc:302] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd1c36a0a9e4fd988a6493dd8e57fc4, a78b175225d64d6490920fe94414dca7; no voters:
I20260807 03:00:58.944320 28192 raft_consensus.cc:2781] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:00:58.944808 28192 raft_consensus.cc:686] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 1 LEADER]: Becoming Leader. State: Replica: 3fd1c36a0a9e4fd988a6493dd8e57fc4, State: Running, Role: LEADER
I20260807 03:00:58.945463 28192 consensus_queue.cc:237] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } }
I20260807 03:00:58.950990 27503 catalog_manager.cc:5495] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fd1c36a0a9e4fd988a6493dd8e57fc4 (127.1.121.65). New cstate: current_term: 1 leader_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fd1c36a0a9e4fd988a6493dd8e57fc4" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37887 } health_report { overall_health: HEALTHY } } }
W20260807 03:00:58.969166 28163 tablet.cc:2369] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:00:58.997254 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:58.991372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.1.121.65:37887)
0807 03:00:58.991600 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:58.991606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:58.991626 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:00:58.991887 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:58.991891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:58.991990 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:58.992157 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:00:58.992596 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.993391 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.994858 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:58.995280 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:58.995417 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:58.996092 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:00:58.996608 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:00:58.996815 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:58.996959 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 03:00:58.997009 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:00:59.005750 28099 raft_consensus.cc:1261] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [term 1 FOLLOWER]: Refusing update from remote peer 3fd1c36a0a9e4fd988a6493dd8e57fc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:00:59.005748 27896 raft_consensus.cc:1261] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [term 1 FOLLOWER]: Refusing update from remote peer 3fd1c36a0a9e4fd988a6493dd8e57fc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:00:59.006803 28192 consensus_queue.cc:1035] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc252cb4508a4815afa3b476af676628" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:00:59.007120 28197 consensus_queue.cc:1035] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a78b175225d64d6490920fe94414dca7" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:00:59.031769 27503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:00:59.032088 27503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:00:59.036927 27503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df4815112170408d99ebd6169fba58cf: Sending DeleteTablet for 3 replicas of tablet 5a44626af202438b92c6a651ce2a14d2
I20260807 03:00:59.038431 28071 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a44626af202438b92c6a651ce2a14d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:59 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:52230
I20260807 03:00:59.038561 27863 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a44626af202438b92c6a651ce2a14d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:59 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:36408
I20260807 03:00:59.038569 27671 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a44626af202438b92c6a651ce2a14d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:59 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:40140
I20260807 03:00:59.039112 27862 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a44626af202438b92c6a651ce2a14d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:00:59 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:36408
I20260807 03:00:59.041106 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 27583
I20260807 03:00:59.049840 28220 tablet_replica.cc:330] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: stopping tablet replica
I20260807 03:00:59.050326 28220 raft_consensus.cc:2229] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:00:59.052095 28220 raft_consensus.cc:2258] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:59.056144 28219 tablet_replica.cc:330] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: stopping tablet replica
I20260807 03:00:59.056555 28219 raft_consensus.cc:2229] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:00:59.057281 28219 raft_consensus.cc:2258] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:59.059247 28219 ts_tablet_manager.cc:1905] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:59.065752 28219 ts_tablet_manager.cc:1918] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:59.066028 28219 log.cc:1196] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/5a44626af202438b92c6a651ce2a14d2
I20260807 03:00:59.066637 28220 ts_tablet_manager.cc:1905] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:59.066908 28219 ts_tablet_manager.cc:1939] T 5a44626af202438b92c6a651ce2a14d2 P 3fd1c36a0a9e4fd988a6493dd8e57fc4: Deleting consensus metadata
I20260807 03:00:59.069476 27474 catalog_manager.cc:4841] TS 3fd1c36a0a9e4fd988a6493dd8e57fc4 (127.1.121.65:37887): tablet 5a44626af202438b92c6a651ce2a14d2 (table test-table [id=827c17502ecd4157994d5450b997fb57]) successfully deleted
I20260807 03:00:59.070883 28218 tablet_replica.cc:330] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: stopping tablet replica
I20260807 03:00:59.071467 28218 raft_consensus.cc:2229] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:00:59.072139 28218 raft_consensus.cc:2258] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:00:59.074626 28218 ts_tablet_manager.cc:1905] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:00:59.075313 28220 ts_tablet_manager.cc:1918] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:59.075610 28220 log.cc:1196] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/5a44626af202438b92c6a651ce2a14d2
I20260807 03:00:59.076427 28220 ts_tablet_manager.cc:1939] T 5a44626af202438b92c6a651ce2a14d2 P a78b175225d64d6490920fe94414dca7: Deleting consensus metadata
I20260807 03:00:59.079036 27479 catalog_manager.cc:4841] TS a78b175225d64d6490920fe94414dca7 (127.1.121.66:33127): tablet 5a44626af202438b92c6a651ce2a14d2 (table test-table [id=827c17502ecd4157994d5450b997fb57]) successfully deleted
W20260807 03:00:59.080592 27479 catalog_manager.cc:4808] TS a78b175225d64d6490920fe94414dca7 (127.1.121.66:33127): delete failed for tablet 5a44626af202438b92c6a651ce2a14d2 because the tablet was not found. No further retry: Not found: Tablet not found: 5a44626af202438b92c6a651ce2a14d2
Aug 07 03:00:59 dist-test-slave-3h5h krb5kdc[27434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 03:00:59.081784 28218 ts_tablet_manager.cc:1918] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:00:59.081954 28218 log.cc:1196] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/5a44626af202438b92c6a651ce2a14d2
I20260807 03:00:59.082577 28218 ts_tablet_manager.cc:1939] T 5a44626af202438b92c6a651ce2a14d2 P fc252cb4508a4815afa3b476af676628: Deleting consensus metadata
I20260807 03:00:59.084412 27473 catalog_manager.cc:4841] TS fc252cb4508a4815afa3b476af676628 (127.1.121.67:37803): tablet 5a44626af202438b92c6a651ce2a14d2 (table test-table [id=827c17502ecd4157994d5450b997fb57]) successfully deleted
I20260807 03:00:59.089956 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:59.042638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.1.121.65:37887)
0807 03:00:59.042834 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:59.042845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:59.067606 (+ 24761us) server_negotiation.cc:407] Connection header received
0807 03:00:59.068210 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:59.068216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:59.068304 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:59.068386 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:59.070402 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:59.071743 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:59.074482 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:59.075363 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:59.075513 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:59.082368 (+ 6855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:59.082387 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:59.082402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:59.082460 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:59.086574 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:59.087614 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:59.087622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:59.087625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:59.087801 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:59.088238 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:59.088246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:59.088254 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:59.088605 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:59.088974 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:59.089321 (+ 347us) server_negotiation.cc:299] Negotiation successful
0807 03:00:59.089607 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 03:00:59.090713 27760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:59.218791 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 27583
I20260807 03:00:59.248395 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 27776
Aug 07 03:00:59 dist-test-slave-3h5h krb5kdc[27434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 03:00:59.264660 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:59.249339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.1.121.66:33127)
0807 03:00:59.249502 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:59.249507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:59.249525 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:00:59.249657 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:59.249661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:59.249735 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:59.249813 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:59.251193 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:59.252042 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:59.253572 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:59.253961 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:59.254062 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:59.258453 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:59.258468 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:59.258486 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:59.258520 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:59.261719 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:59.262466 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:59.262474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:59.262476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:59.262627 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:59.262956 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:59.262959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:59.262961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:59.263395 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:59.263666 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:59.263990 (+ 324us) server_negotiation.cc:299] Negotiation successful
0807 03:00:59.264313 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 03:00:59.265426 27960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:59.360963 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 27776
I20260807 03:00:59.388720 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 27975
Aug 07 03:00:59 dist-test-slave-3h5h krb5kdc[27434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 03:00:59.404526 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:00:59.389633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.1.121.67:37803)
0807 03:00:59.389809 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 03:00:59.389813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:00:59.389851 (+ 38us) server_negotiation.cc:407] Connection header received
0807 03:00:59.390100 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:00:59.390104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:00:59.390176 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:00:59.390265 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:00:59.391836 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:59.392562 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:59.393996 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:00:59.394389 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:00:59.394484 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:00:59.399248 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:00:59.399265 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:00:59.399284 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:00:59.399325 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:00:59.402202 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:59.402824 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:59.402832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:59.402834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:59.403015 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:00:59.403332 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:00:59.403335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:00:59.403337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:00:59.403674 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:00:59.403890 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 03:00:59.404096 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 03:00:59.404265 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 03:00:59.405391 28151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:59.494563 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 27975
I20260807 03:00:59.520334 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 27451
I20260807 03:00:59.521539 27541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:00:59.635563 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 27451
2026-08-07T03:00:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4946 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T03:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:00:59Z Disabled control of system clock
I20260807 03:00:59.738811 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:35239
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:01:00.102869 28231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:00.103214 28231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:00.103310 28231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:00.110802 28231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:00.110941 28231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:00.110988 28231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:00.111048 28231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:00.111097 28231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:00.120926 28231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:35239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:00.122280 28231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:00.124161 28231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:00.136628 28243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:00.136799 28239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:00.136799 28240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:00.137984 28231 server_base.cc:1034] running on GCE node
I20260807 03:01:00.138823 28231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:00.140321 28231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:00.141613 28231 hybrid_clock.cc:648] HybridClock initialized: now 1786071660141561 us; error 56 us; skew 500 ppm
I20260807 03:01:00.142012 28231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:01:00.144361 28231 webserver.cc:460] Webserver started at http://127.1.121.126:45099/ using document root <none> and password file <none>
I20260807 03:01:00.145145 28231 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:00.145292 28231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:00.145627 28231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:00.148590 28231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "c61040243bc644cdb567339f0749d0d2"
format_stamp: "Formatted at 2026-08-07 03:01:00 on dist-test-slave-3h5h"
server_key: "e3714a0fe3093b3220f27190cfa373fd"
server_key_iv: "10b3627807a802a689aaa98513a677fa"
server_key_version: "encryptionkey@0"
I20260807 03:01:00.149475 28231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "c61040243bc644cdb567339f0749d0d2"
format_stamp: "Formatted at 2026-08-07 03:01:00 on dist-test-slave-3h5h"
server_key: "e3714a0fe3093b3220f27190cfa373fd"
server_key_iv: "10b3627807a802a689aaa98513a677fa"
server_key_version: "encryptionkey@0"
I20260807 03:01:00.154774 28231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 03:01:00.160178 28250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:00.162261 28231 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 03:01:00.162470 28231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "c61040243bc644cdb567339f0749d0d2"
format_stamp: "Formatted at 2026-08-07 03:01:00 on dist-test-slave-3h5h"
server_key: "e3714a0fe3093b3220f27190cfa373fd"
server_key_iv: "10b3627807a802a689aaa98513a677fa"
server_key_version: "encryptionkey@0"
I20260807 03:01:00.162776 28231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:00.183051 28231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:00.183918 28231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:00.184180 28231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:00.203635 28231 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:35239
I20260807 03:01:00.203668 28314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:35239 every 8 connection(s)
I20260807 03:01:00.205122 28231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:00.208894 28315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:00.211817 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 28231
I20260807 03:01:00.212004 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:00.212571 1509 external_mini_cluster.cc:1428] Setting key c95b6025c92311180ad85bbae58959d7
I20260807 03:01:00.221316 28315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2: Bootstrap starting.
I20260807 03:01:00.224673 28315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:00.225953 28315 log.cc:824] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:00.226330 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:00.214835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.1.121.126:35239)
0807 03:01:00.215805 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:00.215815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:00.215854 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:01:00.217575 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:00.217598 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:00.218041 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:00.218456 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:00.219052 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:00.220332 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:00.221894 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:00.222546 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:00.222771 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:00.224000 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:00.224008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:00.224022 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:00.224540 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:00.224642 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:00.224667 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:00.224814 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:00.225025 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 03:01:00.225343 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":352,"threads_started":1}
I20260807 03:01:00.229166 28315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2: No bootstrap required, opened a new log
I20260807 03:01:00.236817 28315 raft_consensus.cc:348] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61040243bc644cdb567339f0749d0d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35239 } }
I20260807 03:01:00.237339 28315 raft_consensus.cc:374] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:00.237466 28315 raft_consensus.cc:729] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61040243bc644cdb567339f0749d0d2, State: Initialized, Role: FOLLOWER
I20260807 03:01:00.238106 28315 consensus_queue.cc:260] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61040243bc644cdb567339f0749d0d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35239 } }
I20260807 03:01:00.238358 28315 raft_consensus.cc:388] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:00.238500 28315 raft_consensus.cc:482] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:00.238662 28315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:00.240904 28315 raft_consensus.cc:504] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61040243bc644cdb567339f0749d0d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35239 } }
I20260807 03:01:00.241436 28315 leader_election.cc:302] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c61040243bc644cdb567339f0749d0d2; no voters:
I20260807 03:01:00.242045 28315 leader_election.cc:288] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:00.242192 28324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:00.243019 28324 raft_consensus.cc:686] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [term 1 LEADER]: Becoming Leader. State: Replica: c61040243bc644cdb567339f0749d0d2, State: Running, Role: LEADER
I20260807 03:01:00.243611 28324 consensus_queue.cc:237] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61040243bc644cdb567339f0749d0d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35239 } }
I20260807 03:01:00.244390 28315 sys_catalog.cc:564] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:00.246940 28325 sys_catalog.cc:455] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c61040243bc644cdb567339f0749d0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61040243bc644cdb567339f0749d0d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35239 } } }
I20260807 03:01:00.247416 28325 sys_catalog.cc:458] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:00.246937 28327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c61040243bc644cdb567339f0749d0d2. Latest consensus state: current_term: 1 leader_uuid: "c61040243bc644cdb567339f0749d0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61040243bc644cdb567339f0749d0d2" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 35239 } } }
I20260807 03:01:00.247687 28327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:00.249887 28337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:00.256372 28337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:00.265882 28337 catalog_manager.cc:1347] Generated new cluster ID: 625886d4541440ddb0cb0e20b99dc94c
I20260807 03:01:00.266011 28337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:00.279417 28337 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:00.281278 28337 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:00.297186 28337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2: Generated new TSK 0
I20260807 03:01:00.298118 28337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:00.309572 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35239
--builtin_ntp_servers=127.1.121.84:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:01:00.697365 28353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:00.697649 28353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:00.697736 28353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:00.705178 28353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:00.705288 28353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:00.705394 28353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 03:01:00.714561 28353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:00.715838 28353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:00.717628 28353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:00.731357 28359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:00.731639 28362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:00.731761 28358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:00.732937 28353 server_base.cc:1034] running on GCE node
I20260807 03:01:00.733567 28353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:00.734598 28353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:00.735812 28353 hybrid_clock.cc:648] HybridClock initialized: now 1786071660735759 us; error 49 us; skew 500 ppm
I20260807 03:01:00.736155 28353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:01:00.738698 28353 webserver.cc:460] Webserver started at http://127.1.121.65:36569/ using document root <none> and password file <none>
I20260807 03:01:00.739427 28353 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:00.739549 28353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:00.739888 28353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:00.742869 28353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "39396a856d464290a92b2115efa68243"
format_stamp: "Formatted at 2026-08-07 03:01:00 on dist-test-slave-3h5h"
server_key: "4b8115003d91a08c320eb769c54bb75f"
server_key_iv: "4e2c7b7728df7e75dee376c146183fa0"
server_key_version: "encryptionkey@0"
I20260807 03:01:00.743669 28353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "39396a856d464290a92b2115efa68243"
format_stamp: "Formatted at 2026-08-07 03:01:00 on dist-test-slave-3h5h"
server_key: "4b8115003d91a08c320eb769c54bb75f"
server_key_iv: "4e2c7b7728df7e75dee376c146183fa0"
server_key_version: "encryptionkey@0"
I20260807 03:01:00.750205 28353 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 03:01:00.753909 28369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:00.755270 28353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:01:00.755460 28353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "39396a856d464290a92b2115efa68243"
format_stamp: "Formatted at 2026-08-07 03:01:00 on dist-test-slave-3h5h"
server_key: "4b8115003d91a08c320eb769c54bb75f"
server_key_iv: "4e2c7b7728df7e75dee376c146183fa0"
server_key_version: "encryptionkey@0"
I20260807 03:01:00.755690 28353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:00.771169 28353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:00.771987 28353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:00.772316 28353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:00.773388 28353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:01:00.774904 28353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:01:00.775048 28353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:00.775184 28353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:01:00.775276 28353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:00.809798 28353 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:38551
I20260807 03:01:00.809855 28529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:38551 every 8 connection(s)
I20260807 03:01:00.811720 28353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 03:01:00.819162 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 28353
I20260807 03:01:00.819350 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 03:01:00.819844 1509 external_mini_cluster.cc:1428] Setting key 61ab3f2a17bb8aa618249d43ef619d75
I20260807 03:01:00.822849 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35239
--builtin_ntp_servers=127.1.121.84:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:01:00.825678 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:00.814561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:54527 (local address 127.1.121.126:35239)
0807 03:01:00.815027 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:00.815031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:00.816231 (+ 1200us) server_negotiation.cc:407] Connection header received
0807 03:01:00.818178 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:00.818184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:00.818279 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:00.818405 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:00.819618 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:00.820461 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:00.822545 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:00.822956 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:00.823101 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:00.824648 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:00.824658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:00.824667 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:00.824693 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:00.824747 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:00.824767 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:00.824905 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:00.825126 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 03:01:00.825240 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260807 03:01:00.826512 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:00.815153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:35239 (local address 127.1.121.65:54527)
0807 03:01:00.816059 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:00.816103 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:00.817885 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:00.818655 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:00.818663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:00.818759 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:00.819425 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:00.819454 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:00.820667 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:00.820672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:00.822387 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:00.822403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:00.823356 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:00.823361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:00.823569 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:00.824315 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:01:00.824318 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:00.824359 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:01:00.824437 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:00.824916 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:00.824926 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:00.824932 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 03:01:00.825082 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 03:01:00.825396 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":283,"threads_started":1}
I20260807 03:01:00.828112 28530 heartbeater.cc:347] Connected to a master server at 127.1.121.126:35239
I20260807 03:01:00.828447 28530 heartbeater.cc:464] Registering TS with master...
I20260807 03:01:00.829291 28530 heartbeater.cc:511] Master 127.1.121.126:35239 requested a full tablet report, sending...
I20260807 03:01:00.831378 28271 ts_manager.cc:194] Registered new tserver with Master: 39396a856d464290a92b2115efa68243 (127.1.121.65:38551)
I20260807 03:01:00.832938 28271 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.65:54527
W20260807 03:01:01.183590 28535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:01.183885 28535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:01.183970 28535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:01.191588 28535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:01.191705 28535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:01.191913 28535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 03:01:01.201301 28535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:01.202687 28535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:01.204651 28535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:01.218148 28544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:01.218295 28543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:01.218797 28546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:01.219225 28535 server_base.cc:1034] running on GCE node
I20260807 03:01:01.219789 28535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:01.220753 28535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:01.221956 28535 hybrid_clock.cc:648] HybridClock initialized: now 1786071661221892 us; error 60 us; skew 500 ppm
I20260807 03:01:01.222297 28535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:01:01.224583 28535 webserver.cc:460] Webserver started at http://127.1.121.66:33757/ using document root <none> and password file <none>
I20260807 03:01:01.225373 28535 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:01.225497 28535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:01.225841 28535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:01.228806 28535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "c88c1a4d85b44f0fbc4742c833317ba9"
format_stamp: "Formatted at 2026-08-07 03:01:01 on dist-test-slave-3h5h"
server_key: "e10cde950fe387bece9c8a2971439feb"
server_key_iv: "4ffddfa0479e90d6b6486a79ad1f955f"
server_key_version: "encryptionkey@0"
I20260807 03:01:01.229712 28535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "c88c1a4d85b44f0fbc4742c833317ba9"
format_stamp: "Formatted at 2026-08-07 03:01:01 on dist-test-slave-3h5h"
server_key: "e10cde950fe387bece9c8a2971439feb"
server_key_iv: "4ffddfa0479e90d6b6486a79ad1f955f"
server_key_version: "encryptionkey@0"
I20260807 03:01:01.234973 28535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 03:01:01.238572 28558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:01.239941 28535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:01:01.240130 28535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "c88c1a4d85b44f0fbc4742c833317ba9"
format_stamp: "Formatted at 2026-08-07 03:01:01 on dist-test-slave-3h5h"
server_key: "e10cde950fe387bece9c8a2971439feb"
server_key_iv: "4ffddfa0479e90d6b6486a79ad1f955f"
server_key_version: "encryptionkey@0"
I20260807 03:01:01.240371 28535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:01.270157 28535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:01.271083 28535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:01.271420 28535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:01.272475 28535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:01:01.274099 28535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:01:01.274241 28535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:01.274384 28535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:01:01.274477 28535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:01.316200 28535 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:33031
I20260807 03:01:01.316255 28719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:33031 every 8 connection(s)
I20260807 03:01:01.317622 28535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 03:01:01.318332 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 28535
I20260807 03:01:01.318516 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 03:01:01.319113 1509 external_mini_cluster.cc:1428] Setting key cb26f4bf25c9ad94e4b6a0035b69b5c1
I20260807 03:01:01.322559 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35239
--builtin_ntp_servers=127.1.121.84:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:01:01.332126 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.320758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:33105 (local address 127.1.121.126:35239)
0807 03:01:01.320989 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:01.320995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:01.322602 (+ 1607us) server_negotiation.cc:407] Connection header received
0807 03:01:01.324502 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:01.324507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:01.324602 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:01.324722 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:01.325954 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.327075 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.328688 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.329425 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.329574 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.331169 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:01.331179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:01.331186 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:01.331213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:01.331257 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:01.331277 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:01.331371 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:01.331613 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 03:01:01.331727 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 03:01:01.333225 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.321450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:35239 (local address 127.1.121.66:33105)
0807 03:01:01.322439 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:01.322486 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:01.324231 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:01.324990 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:01.324998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:01.325085 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:01.325747 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.325775 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.327272 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.327278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.328532 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.328544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.329661 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.329678 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.329981 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.330791 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:01:01.330796 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:01.330848 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:01:01.330943 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:01.331381 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:01.331392 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:01.331403 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 03:01:01.331572 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 03:01:01.331963 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":347,"threads_started":1}
I20260807 03:01:01.336089 28721 heartbeater.cc:347] Connected to a master server at 127.1.121.126:35239
I20260807 03:01:01.336414 28721 heartbeater.cc:464] Registering TS with master...
I20260807 03:01:01.337468 28721 heartbeater.cc:511] Master 127.1.121.126:35239 requested a full tablet report, sending...
I20260807 03:01:01.339133 28271 ts_manager.cc:194] Registered new tserver with Master: c88c1a4d85b44f0fbc4742c833317ba9 (127.1.121.66:33031)
I20260807 03:01:01.339999 28271 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.66:33105
W20260807 03:01:01.677287 28727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:01.677572 28727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:01.677749 28727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:01.685001 28727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:01.685112 28727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:01.685329 28727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 03:01:01.694461 28727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:35239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:01.695797 28727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:01.697674 28727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:01.711221 28734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:01.711326 28735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:01.711295 28739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:01.712330 28727 server_base.cc:1034] running on GCE node
I20260807 03:01:01.713047 28727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:01.714038 28727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:01.715279 28727 hybrid_clock.cc:648] HybridClock initialized: now 1786071661715241 us; error 61 us; skew 500 ppm
I20260807 03:01:01.715610 28727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:01:01.717932 28727 webserver.cc:460] Webserver started at http://127.1.121.67:36917/ using document root <none> and password file <none>
I20260807 03:01:01.718650 28727 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:01.718794 28727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:01.719130 28727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:01.722126 28727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "277741366f6b4fbe85443156a7df7505"
format_stamp: "Formatted at 2026-08-07 03:01:01 on dist-test-slave-3h5h"
server_key: "483b5be7f8104e7d11acf39ff1923706"
server_key_iv: "865fb2a4a297e5577a771ab28298a1a7"
server_key_version: "encryptionkey@0"
I20260807 03:01:01.722949 28727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "277741366f6b4fbe85443156a7df7505"
format_stamp: "Formatted at 2026-08-07 03:01:01 on dist-test-slave-3h5h"
server_key: "483b5be7f8104e7d11acf39ff1923706"
server_key_iv: "865fb2a4a297e5577a771ab28298a1a7"
server_key_version: "encryptionkey@0"
I20260807 03:01:01.728236 28727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260807 03:01:01.731751 28746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:01.733146 28727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:01:01.733347 28727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "277741366f6b4fbe85443156a7df7505"
format_stamp: "Formatted at 2026-08-07 03:01:01 on dist-test-slave-3h5h"
server_key: "483b5be7f8104e7d11acf39ff1923706"
server_key_iv: "865fb2a4a297e5577a771ab28298a1a7"
server_key_version: "encryptionkey@0"
I20260807 03:01:01.733578 28727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:01.749711 28727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:01.750965 28727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:01.751355 28727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:01.752622 28727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:01:01.754230 28727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:01:01.754320 28727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:01.754464 28727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:01:01.754561 28727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:01.788941 28727 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:33275
I20260807 03:01:01.789072 28905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:33275 every 8 connection(s)
I20260807 03:01:01.790360 28727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 03:01:01.799052 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 28727
I20260807 03:01:01.799285 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 03:01:01.799728 1509 external_mini_cluster.cc:1428] Setting key 621171cdd23a64573b86d9b5dbb81d2c
I20260807 03:01:01.803342 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.792993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:40341 (local address 127.1.121.126:35239)
0807 03:01:01.793175 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:01.793180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:01.794501 (+ 1321us) server_negotiation.cc:407] Connection header received
0807 03:01:01.796341 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:01.796347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:01.796444 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:01.796559 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:01.797658 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.798441 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.800185 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.800779 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.800957 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.802473 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:01.802479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:01.802486 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:01.802511 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:01.802543 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:01.802555 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:01.802654 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:01.802928 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 03:01:01.803009 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 03:01:01.804304 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.793413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:35239 (local address 127.1.121.67:40341)
0807 03:01:01.794323 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:01.794368 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:01.796052 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:01.796804 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:01.796812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:01.796912 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:01.797479 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.797496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.798607 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.798614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.800030 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.800042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.801037 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.801046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.801351 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.802149 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:01:01.802153 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:01.802196 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:01:01.802275 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:01.802671 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:01.802684 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:01.802694 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 03:01:01.802898 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 03:01:01.803265 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":308,"threads_started":1}
I20260807 03:01:01.805805 28906 heartbeater.cc:347] Connected to a master server at 127.1.121.126:35239
I20260807 03:01:01.806113 28906 heartbeater.cc:464] Registering TS with master...
I20260807 03:01:01.806921 28906 heartbeater.cc:511] Master 127.1.121.126:35239 requested a full tablet report, sending...
I20260807 03:01:01.808347 28271 ts_manager.cc:194] Registered new tserver with Master: 277741366f6b4fbe85443156a7df7505 (127.1.121.67:33275)
I20260807 03:01:01.809248 28271 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.67:40341
I20260807 03:01:01.815513 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:01:01.826108 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.819812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.1.121.126:35239)
0807 03:01:01.820053 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:01.820061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:01.820403 (+ 342us) server_negotiation.cc:407] Connection header received
0807 03:01:01.820610 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:01.820615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:01.820703 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:01.820812 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:01.821357 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.822054 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.823432 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.823895 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.824034 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.825325 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:01.825332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:01.825337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:01.825363 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:01.825396 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:01.825409 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:01.825565 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:01.825768 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 03:01:01.825826 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:01:01.830024 28271 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 03:01:01.835937 28530 heartbeater.cc:503] Master 127.1.121.126:35239 was elected leader, sending a full tablet report...
W20260807 03:01:01.837566 28271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:01:01.872298 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.849928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.1.121.67:33275)
0807 03:01:01.862352 (+ 12424us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:01.862360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:01.862396 (+ 36us) server_negotiation.cc:407] Connection header received
0807 03:01:01.862505 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:01.862514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:01.862834 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:01.863140 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:01.864271 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.865973 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.869033 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.870528 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.870661 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.870949 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:01.871667 (+ 718us) server_negotiation.cc:299] Negotiation successful
0807 03:01:01.871856 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12231,"thread_start_us":12072,"threads_started":1}
I20260807 03:01:01.872681 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.862362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42370 (local address 127.1.121.66:33031)
0807 03:01:01.863091 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:01.863103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:01.864682 (+ 1579us) server_negotiation.cc:407] Connection header received
0807 03:01:01.864907 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:01.864915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:01.865127 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:01.865337 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:01.865911 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.867073 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.869937 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.871269 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.871412 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.871641 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:01.872179 (+ 538us) server_negotiation.cc:299] Negotiation successful
0807 03:01:01.872346 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":375,"threads_started":1}
I20260807 03:01:01.872931 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.850115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38551 (local address 127.0.0.1:55968)
0807 03:01:01.863750 (+ 13635us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:01.863773 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:01.864054 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:01.865073 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:01.865075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:01.865095 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:01.865483 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.865495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.867440 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.867446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.869875 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.869893 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.872078 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.872084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.872267 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.872310 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 03:01:01.872394 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 03:01:01.872531 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13450,"mutex_wait_us":3,"thread_start_us":11107,"threads_started":1}
I20260807 03:01:01.873077 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.849663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:33275 (local address 127.0.0.1:40094)
0807 03:01:01.860627 (+ 10964us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:01.860695 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:01.860958 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:01.863523 (+ 2565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:01.863530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:01.863577 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:01.864027 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.864040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.866176 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.866182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.868735 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.868751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.871091 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.871098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.871380 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.871437 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 03:01:01.872588 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0807 03:01:01.872712 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10569,"mutex_wait_us":34,"thread_start_us":15153,"threads_started":1}
I20260807 03:01:01.873610 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.863070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:33031 (local address 127.0.0.1:42370)
0807 03:01:01.864564 (+ 1494us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:01.864582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:01.864710 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:01.865349 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:01.865354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:01.865382 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:01.865744 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.865753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.867254 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.867259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.869706 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:01.869723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.871779 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.871784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:01.871953 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.871976 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 03:01:01.872113 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 03:01:01.872324 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1370,"mutex_wait_us":4,"thread_start_us":318,"threads_started":1}
I20260807 03:01:01.873801 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:01.850706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.1.121.65:38551)
0807 03:01:01.861626 (+ 10920us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:01.861636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:01.863904 (+ 2268us) server_negotiation.cc:407] Connection header received
0807 03:01:01.864270 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:01.864281 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:01.864521 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:01.864713 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:01.865641 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.867264 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.870080 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:01.871498 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:01.871636 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:01.871940 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:01.873199 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0807 03:01:01.873388 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10731,"thread_start_us":10929,"threads_started":1}
I20260807 03:01:01.877278 28624 tablet_service.cc:1467] Processing CreateTablet for tablet ae38f0330d2c46a7980a5d84d029e1a2 (DEFAULT_TABLE table=test-table [id=7efb08f8d8b74ca8b646cb8fa9829f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:01:01.877331 28440 tablet_service.cc:1467] Processing CreateTablet for tablet ae38f0330d2c46a7980a5d84d029e1a2 (DEFAULT_TABLE table=test-table [id=7efb08f8d8b74ca8b646cb8fa9829f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:01:01.877329 28815 tablet_service.cc:1467] Processing CreateTablet for tablet ae38f0330d2c46a7980a5d84d029e1a2 (DEFAULT_TABLE table=test-table [id=7efb08f8d8b74ca8b646cb8fa9829f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:01:01.878582 28624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae38f0330d2c46a7980a5d84d029e1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:01.878753 28815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae38f0330d2c46a7980a5d84d029e1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:01.878770 28440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae38f0330d2c46a7980a5d84d029e1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:01.892231 28933 tablet_bootstrap.cc:492] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: Bootstrap starting.
I20260807 03:01:01.896521 28933 tablet_bootstrap.cc:654] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:01.897671 28935 tablet_bootstrap.cc:492] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: Bootstrap starting.
I20260807 03:01:01.898026 28933 log.cc:824] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:01.900437 28934 tablet_bootstrap.cc:492] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: Bootstrap starting.
I20260807 03:01:01.901795 28933 tablet_bootstrap.cc:492] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: No bootstrap required, opened a new log
I20260807 03:01:01.901875 28935 tablet_bootstrap.cc:654] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:01.902094 28933 ts_tablet_manager.cc:1397] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260807 03:01:01.903167 28935 log.cc:824] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:01.903473 28934 tablet_bootstrap.cc:654] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:01.904584 28934 log.cc:824] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:01.907573 28935 tablet_bootstrap.cc:492] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: No bootstrap required, opened a new log
I20260807 03:01:01.907829 28934 tablet_bootstrap.cc:492] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: No bootstrap required, opened a new log
I20260807 03:01:01.907863 28935 ts_tablet_manager.cc:1397] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260807 03:01:01.908104 28934 ts_tablet_manager.cc:1397] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260807 03:01:01.913458 28933 raft_consensus.cc:348] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } }
I20260807 03:01:01.913992 28933 raft_consensus.cc:374] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:01.914103 28933 raft_consensus.cc:729] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88c1a4d85b44f0fbc4742c833317ba9, State: Initialized, Role: FOLLOWER
I20260807 03:01:01.914902 28933 consensus_queue.cc:260] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } }
I20260807 03:01:01.915611 28935 raft_consensus.cc:348] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } }
I20260807 03:01:01.916079 28935 raft_consensus.cc:374] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:01.916198 28935 raft_consensus.cc:729] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277741366f6b4fbe85443156a7df7505, State: Initialized, Role: FOLLOWER
I20260807 03:01:01.916826 28721 heartbeater.cc:503] Master 127.1.121.126:35239 was elected leader, sending a full tablet report...
I20260807 03:01:01.916920 28935 consensus_queue.cc:260] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } }
I20260807 03:01:01.917445 28933 ts_tablet_manager.cc:1428] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260807 03:01:01.918507 28906 heartbeater.cc:503] Master 127.1.121.126:35239 was elected leader, sending a full tablet report...
I20260807 03:01:01.919111 28935 ts_tablet_manager.cc:1428] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 03:01:01.919569 28934 raft_consensus.cc:348] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } }
I20260807 03:01:01.920080 28934 raft_consensus.cc:374] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:01.920195 28934 raft_consensus.cc:729] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39396a856d464290a92b2115efa68243, State: Initialized, Role: FOLLOWER
I20260807 03:01:01.920756 28934 consensus_queue.cc:260] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } }
I20260807 03:01:01.922545 28934 ts_tablet_manager.cc:1428] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260807 03:01:02.042120 28907 tablet.cc:2369] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:01:02.064352 28531 tablet.cc:2369] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:01:02.069885 28724 tablet.cc:2369] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:01:02.243296 28946 raft_consensus.cc:482] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:01:02.243631 28946 raft_consensus.cc:504] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } }
I20260807 03:01:02.245793 28946 leader_election.cc:288] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39396a856d464290a92b2115efa68243 (127.1.121.65:38551), c88c1a4d85b44f0fbc4742c833317ba9 (127.1.121.66:33031)
I20260807 03:01:02.251959 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:02.246241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:53611 (local address 127.1.121.65:38551)
0807 03:01:02.246784 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:02.246790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:02.246810 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:01:02.246889 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:02.246894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:02.247015 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:02.247121 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:02.247751 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.248438 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.250250 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.251059 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.251247 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:02.251319 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:02.251581 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 03:01:02.251707 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
I20260807 03:01:02.251959 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:02.246235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38551 (local address 127.1.121.67:53611)
0807 03:01:02.246410 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:02.246426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:02.246529 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:02.247315 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:02.247324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:02.247352 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:02.247585 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:02.247596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.248568 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.248575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:02.250039 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:02.250050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.251237 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.251243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:02.251420 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:02.251443 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 03:01:02.251550 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 03:01:02.251695 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 03:01:02.253088 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:02.246696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:33031 (local address 127.1.121.67:58633)
0807 03:01:02.247336 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:02.247352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:02.247473 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:02.247839 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:02.247844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:02.247871 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:02.248241 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:02.248251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.249410 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.249416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:02.251126 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:02.251140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.252442 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.252448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:02.252602 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:02.252624 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:01:02.252702 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 03:01:02.252781 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"spinlock_wait_cycles":896,"thread_start_us":348,"threads_started":1}
I20260807 03:01:02.253304 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:02.246847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:58633 (local address 127.1.121.66:33031)
0807 03:01:02.247120 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:02.247127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:02.247443 (+ 316us) server_negotiation.cc:407] Connection header received
0807 03:01:02.247609 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:02.247613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:02.247690 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:02.247820 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:02.248431 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.249245 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.251326 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.252275 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.252458 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:02.252561 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:02.252772 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 03:01:02.252971 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 03:01:02.253157 28463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae38f0330d2c46a7980a5d84d029e1a2" candidate_uuid: "277741366f6b4fbe85443156a7df7505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39396a856d464290a92b2115efa68243" is_pre_election: true
I20260807 03:01:02.253883 28463 raft_consensus.cc:2454] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277741366f6b4fbe85443156a7df7505 in term 0.
I20260807 03:01:02.254333 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae38f0330d2c46a7980a5d84d029e1a2" candidate_uuid: "277741366f6b4fbe85443156a7df7505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" is_pre_election: true
I20260807 03:01:02.254705 28752 leader_election.cc:302] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277741366f6b4fbe85443156a7df7505, 39396a856d464290a92b2115efa68243; no voters:
I20260807 03:01:02.254966 28654 raft_consensus.cc:2454] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277741366f6b4fbe85443156a7df7505 in term 0.
I20260807 03:01:02.255098 28946 raft_consensus.cc:2781] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:01:02.255277 28946 raft_consensus.cc:482] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:01:02.255370 28946 raft_consensus.cc:3037] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:02.257664 28946 raft_consensus.cc:504] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } }
I20260807 03:01:02.258203 28946 leader_election.cc:288] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [CANDIDATE]: Term 1 election: Requested vote from peers 39396a856d464290a92b2115efa68243 (127.1.121.65:38551), c88c1a4d85b44f0fbc4742c833317ba9 (127.1.121.66:33031)
I20260807 03:01:02.258790 28463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae38f0330d2c46a7980a5d84d029e1a2" candidate_uuid: "277741366f6b4fbe85443156a7df7505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39396a856d464290a92b2115efa68243"
I20260807 03:01:02.258858 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae38f0330d2c46a7980a5d84d029e1a2" candidate_uuid: "277741366f6b4fbe85443156a7df7505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88c1a4d85b44f0fbc4742c833317ba9"
I20260807 03:01:02.259055 28463 raft_consensus.cc:3037] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:02.259193 28654 raft_consensus.cc:3037] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:02.262799 28463 raft_consensus.cc:2454] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277741366f6b4fbe85443156a7df7505 in term 1.
I20260807 03:01:02.262813 28654 raft_consensus.cc:2454] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277741366f6b4fbe85443156a7df7505 in term 1.
I20260807 03:01:02.263291 28752 leader_election.cc:302] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277741366f6b4fbe85443156a7df7505, c88c1a4d85b44f0fbc4742c833317ba9; no voters:
I20260807 03:01:02.263623 28946 raft_consensus.cc:2781] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:02.264155 28946 raft_consensus.cc:686] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 1 LEADER]: Becoming Leader. State: Replica: 277741366f6b4fbe85443156a7df7505, State: Running, Role: LEADER
I20260807 03:01:02.264777 28946 consensus_queue.cc:237] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } }
I20260807 03:01:02.270048 28270 catalog_manager.cc:5495] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 reported cstate change: term changed from 0 to 1, leader changed from <none> to 277741366f6b4fbe85443156a7df7505 (127.1.121.67). New cstate: current_term: 1 leader_uuid: "277741366f6b4fbe85443156a7df7505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277741366f6b4fbe85443156a7df7505" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 33275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 } health_report { overall_health: UNKNOWN } } }
I20260807 03:01:02.334097 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:02.328157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.1.121.67:33275)
0807 03:01:02.328383 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:02.328390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:02.328559 (+ 169us) server_negotiation.cc:407] Connection header received
0807 03:01:02.328805 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:02.328810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:02.328985 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:02.329147 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:01:02.329682 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.330402 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.331846 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.332255 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.332394 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:02.333009 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:01:02.333479 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:01:02.333677 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:02.333827 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 03:01:02.333876 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 03:01:02.342840 28654 raft_consensus.cc:1261] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [term 1 FOLLOWER]: Refusing update from remote peer 277741366f6b4fbe85443156a7df7505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:01:02.342840 28463 raft_consensus.cc:1261] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [term 1 FOLLOWER]: Refusing update from remote peer 277741366f6b4fbe85443156a7df7505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:01:02.343887 28950 consensus_queue.cc:1035] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39396a856d464290a92b2115efa68243" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:01:02.344261 28946 consensus_queue.cc:1035] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c88c1a4d85b44f0fbc4742c833317ba9" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:01:02.354535 28956 mvcc.cc:205] Tried to move back new op lower bound from 7315749528945311744 to 7315749528641601536. Current Snapshot: MvccSnapshot[applied={T|T < 7315749528945311744}]
I20260807 03:01:02.356312 28957 mvcc.cc:205] Tried to move back new op lower bound from 7315749528945311744 to 7315749528641601536. Current Snapshot: MvccSnapshot[applied={T|T < 7315749528945311744}]
I20260807 03:01:02.368625 28270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:01:02.369110 28270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:01:02.373111 28270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c61040243bc644cdb567339f0749d0d2: Sending DeleteTablet for 3 replicas of tablet ae38f0330d2c46a7980a5d84d029e1a2
I20260807 03:01:02.374356 28815 tablet_service.cc:1514] Processing DeleteTablet for tablet ae38f0330d2c46a7980a5d84d029e1a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:01:02 UTC) from {username='slave'} at 127.0.0.1:40094
I20260807 03:01:02.374627 28440 tablet_service.cc:1514] Processing DeleteTablet for tablet ae38f0330d2c46a7980a5d84d029e1a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:01:02 UTC) from {username='slave'} at 127.0.0.1:55968
I20260807 03:01:02.375023 28624 tablet_service.cc:1514] Processing DeleteTablet for tablet ae38f0330d2c46a7980a5d84d029e1a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:01:02 UTC) from {username='slave'} at 127.0.0.1:42370
I20260807 03:01:02.375692 28967 tablet_replica.cc:330] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: stopping tablet replica
I20260807 03:01:02.376056 28968 tablet_replica.cc:330] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: stopping tablet replica
I20260807 03:01:02.376246 28967 raft_consensus.cc:2229] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:01:02.376535 28968 raft_consensus.cc:2229] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:01:02.376741 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 28353
I20260807 03:01:02.377023 28967 raft_consensus.cc:2258] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:01:02.377152 28968 raft_consensus.cc:2258] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:01:02.379868 28968 ts_tablet_manager.cc:1905] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:01:02.381605 28967 ts_tablet_manager.cc:1905] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:01:02.388356 28968 ts_tablet_manager.cc:1918] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:01:02.388588 28968 log.cc:1196] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/ae38f0330d2c46a7980a5d84d029e1a2
I20260807 03:01:02.389516 28968 ts_tablet_manager.cc:1939] T ae38f0330d2c46a7980a5d84d029e1a2 P c88c1a4d85b44f0fbc4742c833317ba9: Deleting consensus metadata
I20260807 03:01:02.390954 28967 ts_tablet_manager.cc:1918] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:01:02.391141 28967 log.cc:1196] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/ae38f0330d2c46a7980a5d84d029e1a2
I20260807 03:01:02.391709 28967 ts_tablet_manager.cc:1939] T ae38f0330d2c46a7980a5d84d029e1a2 P 39396a856d464290a92b2115efa68243: Deleting consensus metadata
I20260807 03:01:02.391757 28255 catalog_manager.cc:4841] TS c88c1a4d85b44f0fbc4742c833317ba9 (127.1.121.66:33031): tablet ae38f0330d2c46a7980a5d84d029e1a2 (table test-table [id=7efb08f8d8b74ca8b646cb8fa9829f1d]) successfully deleted
I20260807 03:01:02.393711 28255 catalog_manager.cc:4841] TS 39396a856d464290a92b2115efa68243 (127.1.121.65:38551): tablet ae38f0330d2c46a7980a5d84d029e1a2 (table test-table [id=7efb08f8d8b74ca8b646cb8fa9829f1d]) successfully deleted
I20260807 03:01:02.394021 28966 tablet_replica.cc:330] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: stopping tablet replica
I20260807 03:01:02.394484 28966 raft_consensus.cc:2229] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:01:02.395079 28966 raft_consensus.cc:2258] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:01:02.396931 28966 ts_tablet_manager.cc:1905] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:01:02.402082 28966 ts_tablet_manager.cc:1918] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:01:02.402266 28966 log.cc:1196] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/ae38f0330d2c46a7980a5d84d029e1a2
I20260807 03:01:02.402734 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:02.379847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.1.121.65:38551)
0807 03:01:02.380048 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:02.380057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:02.393614 (+ 13557us) server_negotiation.cc:407] Connection header received
0807 03:01:02.394080 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:02.394086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:02.394206 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:02.394289 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:02.395092 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.396219 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.398323 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.398997 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.399106 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:02.400781 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:02.400794 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:02.400812 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:02.401669 (+ 857us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:02.401735 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:02.401771 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:02.401865 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:02.402155 (+ 290us) server_negotiation.cc:299] Negotiation successful
0807 03:01:02.402286 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:01:02.402755 28966 ts_tablet_manager.cc:1939] T ae38f0330d2c46a7980a5d84d029e1a2 P 277741366f6b4fbe85443156a7df7505: Deleting consensus metadata
I20260807 03:01:02.403983 28524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:02.404733 28252 catalog_manager.cc:4841] TS 277741366f6b4fbe85443156a7df7505 (127.1.121.67:33275): tablet ae38f0330d2c46a7980a5d84d029e1a2 (table test-table [id=7efb08f8d8b74ca8b646cb8fa9829f1d]) successfully deleted
I20260807 03:01:02.522260 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 28353
I20260807 03:01:02.547529 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 28535
I20260807 03:01:02.560452 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:02.548370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.1.121.66:33031)
0807 03:01:02.548564 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:02.548569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:02.548623 (+ 54us) server_negotiation.cc:407] Connection header received
0807 03:01:02.548888 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:02.548894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:02.548990 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:02.549087 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:02.549603 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.550480 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.552328 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.552735 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.552827 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:02.559024 (+ 6197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:02.559034 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:02.559050 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:02.559642 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:02.559707 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:02.559727 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:02.559822 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:02.559983 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 03:01:02.560075 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 03:01:02.561205 28715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:02.651209 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 28535
I20260807 03:01:02.686321 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 28727
I20260807 03:01:02.693912 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:02.687208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.1.121.67:33275)
0807 03:01:02.687437 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:02.687443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:02.687463 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:01:02.687638 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:02.687643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:02.687756 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:02.687886 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:02.688361 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.689253 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.690522 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:02.690951 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:02.691113 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:02.692249 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:02.692258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:02.692277 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:02.693072 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:02.693138 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:02.693169 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:02.693292 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:02.693427 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 03:01:02.693527 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 03:01:02.694756 28901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:02.818266 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 28727
I20260807 03:01:02.846355 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 28231
I20260807 03:01:02.847572 28309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:02.959125 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 28231
2026-08-07T03:01:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3322 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:03 dist-test-slave-3h5h krb5kdc[28978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:03 dist-test-slave-3h5h krb5kdc[28978](info): set up 2 sockets
Aug 07 03:01:03 dist-test-slave-3h5h krb5kdc[28978](info): commencing operation
krb5kdc: starting...
W20260807 03:01:04.356720 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.345s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:04 dist-test-slave-3h5h krb5kdc[28978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:01:04.700533 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:36453
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:36453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:05.082672 29003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:05.082978 29003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:05.083068 29003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:05.090301 29003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:05.090422 29003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:05.090468 29003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:05.090529 29003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:05.090574 29003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:05.100440 29003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:36453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:36453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:05.101754 29003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:05.103017 29003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:05.115350 29010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:05.115396 29015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:05.115372 29013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:05.118374 29003 server_base.cc:1034] running on GCE node
I20260807 03:01:05.119189 29003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:05.120661 29003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:05.121908 29003 hybrid_clock.cc:648] HybridClock initialized: now 1786071665121867 us; error 49 us; skew 500 ppm
I20260807 03:01:05.122287 29003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:05 dist-test-slave-3h5h krb5kdc[28978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071665, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:05.125473 29003 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:01:05.127128 29003 webserver.cc:460] Webserver started at http://127.1.121.126:33569/ using document root <none> and password file <none>
I20260807 03:01:05.127871 29003 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:05.128021 29003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:05.128355 29003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:05.131435 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "a9e04b1ce5914d25985db3f7caa5808b"
format_stamp: "Formatted at 2026-08-07 03:01:05 on dist-test-slave-3h5h"
server_key: "af7ee95fe40b84fea090983cbdd0c9fa"
server_key_iv: "926a0beb46a183c47592140a1c0e359f"
server_key_version: "encryptionkey@0"
I20260807 03:01:05.132207 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "a9e04b1ce5914d25985db3f7caa5808b"
format_stamp: "Formatted at 2026-08-07 03:01:05 on dist-test-slave-3h5h"
server_key: "af7ee95fe40b84fea090983cbdd0c9fa"
server_key_iv: "926a0beb46a183c47592140a1c0e359f"
server_key_version: "encryptionkey@0"
I20260807 03:01:05.137777 29003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 03:01:05.141409 29025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:05.142769 29003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:01:05.142962 29003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "a9e04b1ce5914d25985db3f7caa5808b"
format_stamp: "Formatted at 2026-08-07 03:01:05 on dist-test-slave-3h5h"
server_key: "af7ee95fe40b84fea090983cbdd0c9fa"
server_key_iv: "926a0beb46a183c47592140a1c0e359f"
server_key_version: "encryptionkey@0"
I20260807 03:01:05.143190 29003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:05.161116 29003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:05.165468 29003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:05.165796 29003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:05.194290 29003 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:36453
I20260807 03:01:05.194293 29098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:36453 every 8 connection(s)
I20260807 03:01:05.195755 29003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:05.199564 29099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:05.204413 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 29003
I20260807 03:01:05.204640 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:05.205240 1509 external_mini_cluster.cc:1428] Setting key 8554c375ce21aed48abab21697fae3d0
I20260807 03:01:05.211715 29099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b: Bootstrap starting.
I20260807 03:01:05.214875 29099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:05.215942 29099 log.cc:824] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:05.218745 29099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b: No bootstrap required, opened a new log
Aug 07 03:01:05 dist-test-slave-3h5h krb5kdc[28978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:01:05.228029 29099 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e04b1ce5914d25985db3f7caa5808b" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36453 } }
I20260807 03:01:05.228416 29099 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:05.228547 29099 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e04b1ce5914d25985db3f7caa5808b, State: Initialized, Role: FOLLOWER
I20260807 03:01:05.229142 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:05.207324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.1.121.126:36453)
0807 03:01:05.208202 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:05.208214 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:05.208260 (+ 46us) server_negotiation.cc:407] Connection header received
0807 03:01:05.210461 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:05.210486 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:05.210873 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:05.211246 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:05.212863 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:05.214185 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:05.215545 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:05.216293 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:05.216506 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:05.221273 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:05.221280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:05.221294 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:05.221337 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:05.224659 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:05.225530 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:05.225536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:05.225543 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:05.225743 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:05.226184 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:05.226187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:05.226189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:05.226759 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:05.226996 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:05.227434 (+ 438us) server_negotiation.cc:299] Negotiation successful
0807 03:01:05.227848 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":404,"threads_started":1}
I20260807 03:01:05.229224 29099 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e04b1ce5914d25985db3f7caa5808b" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36453 } }
I20260807 03:01:05.229555 29099 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:05.229745 29099 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:05.229938 29099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:05.233108 29099 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e04b1ce5914d25985db3f7caa5808b" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36453 } }
I20260807 03:01:05.233587 29099 leader_election.cc:302] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e04b1ce5914d25985db3f7caa5808b; no voters:
I20260807 03:01:05.234159 29099 leader_election.cc:288] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:05.234306 29105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:05.235026 29105 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [term 1 LEADER]: Becoming Leader. State: Replica: a9e04b1ce5914d25985db3f7caa5808b, State: Running, Role: LEADER
I20260807 03:01:05.235612 29105 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e04b1ce5914d25985db3f7caa5808b" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36453 } }
I20260807 03:01:05.236188 29099 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:05.238560 29107 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e04b1ce5914d25985db3f7caa5808b. Latest consensus state: current_term: 1 leader_uuid: "a9e04b1ce5914d25985db3f7caa5808b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e04b1ce5914d25985db3f7caa5808b" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36453 } } }
I20260807 03:01:05.238528 29106 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9e04b1ce5914d25985db3f7caa5808b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e04b1ce5914d25985db3f7caa5808b" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36453 } } }
I20260807 03:01:05.238987 29107 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:05.238987 29106 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:05.242738 29113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:05.248229 29113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:05.257341 29113 catalog_manager.cc:1347] Generated new cluster ID: 9178bd5bba2e479293dbf3aa83a1f7d0
I20260807 03:01:05.257463 29113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:05.281335 29113 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:05.282702 29113 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:05.302912 29113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b: Generated new TSK 0
I20260807 03:01:05.304229 29113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.121.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
Entry for principal kudu/127.1.121.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab.
I20260807 03:01:05.404243 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:36453
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:05.770905 29130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:05.771206 29130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:05.771337 29130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:05.779011 29130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:05.779145 29130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:05.779350 29130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 03:01:05.789696 29130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.65.keytab
--principal=kudu/127.1.121.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:36453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:05.791095 29130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:05.792508 29130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:05.806620 29137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:05.806653 29138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:05.806670 29142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:05.810447 29130 server_base.cc:1034] running on GCE node
I20260807 03:01:05.811106 29130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:05.812124 29130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:05.813382 29130 hybrid_clock.cc:648] HybridClock initialized: now 1786071665813337 us; error 55 us; skew 500 ppm
I20260807 03:01:05.813735 29130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:05 dist-test-slave-3h5h krb5kdc[28978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071665, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:05.817121 29130 init.cc:377] Logged in from keytab as kudu/127.1.121.65@KRBTEST.COM (short username kudu)
I20260807 03:01:05.818627 29130 webserver.cc:460] Webserver started at http://127.1.121.65:42789/ using document root <none> and password file <none>
I20260807 03:01:05.819365 29130 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:05.819470 29130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:05.819739 29130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:05.822798 29130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "bbc306f2a23445c39fee26340fef0b69"
format_stamp: "Formatted at 2026-08-07 03:01:05 on dist-test-slave-3h5h"
server_key: "70676345cbd04369b7874afd6743d742"
server_key_iv: "5c33318f469fc23e741336d41f94a3ae"
server_key_version: "encryptionkey@0"
I20260807 03:01:05.823577 29130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "bbc306f2a23445c39fee26340fef0b69"
format_stamp: "Formatted at 2026-08-07 03:01:05 on dist-test-slave-3h5h"
server_key: "70676345cbd04369b7874afd6743d742"
server_key_iv: "5c33318f469fc23e741336d41f94a3ae"
server_key_version: "encryptionkey@0"
I20260807 03:01:05.828763 29130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 03:01:05.832172 29152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:05.833696 29130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:01:05.833839 29130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "bbc306f2a23445c39fee26340fef0b69"
format_stamp: "Formatted at 2026-08-07 03:01:05 on dist-test-slave-3h5h"
server_key: "70676345cbd04369b7874afd6743d742"
server_key_iv: "5c33318f469fc23e741336d41f94a3ae"
server_key_version: "encryptionkey@0"
I20260807 03:01:05.834007 29130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:05.880568 29130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:05.885429 29130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:05.885713 29130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:05.886701 29130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:01:05.888136 29130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:01:05.888221 29130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:05.888309 29130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:01:05.888356 29130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:05.924253 29130 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:38911
I20260807 03:01:05.924291 29305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:38911 every 8 connection(s)
I20260807 03:01:05.925678 29130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 03:01:05.928968 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 29130
I20260807 03:01:05.929143 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 03:01:05.929759 1509 external_mini_cluster.cc:1428] Setting key 5a4d496fe1fa69439dad60d74d69fd68
Aug 07 03:01:05 dist-test-slave-3h5h krb5kdc[28978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071665, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.65@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:01:05.950212 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:05.928348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:58671 (local address 127.1.121.126:36453)
0807 03:01:05.928933 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:05.928940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:05.930116 (+ 1176us) server_negotiation.cc:407] Connection header received
0807 03:01:05.932396 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:05.932400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:05.932463 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:05.932555 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:05.934741 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:05.935458 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:05.938174 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:05.938932 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:05.939056 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:05.943668 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:05.943674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:05.943677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:05.943716 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:05.946679 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:05.947573 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:05.947578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:05.947580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:05.947711 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:05.948243 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:05.948249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:05.948252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:05.948549 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:05.948719 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:05.949567 (+ 848us) server_negotiation.cc:299] Negotiation successful
0807 03:01:05.949812 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":250,"threads_started":1}
I20260807 03:01:05.951284 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:05.928987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:36453 (local address 127.1.121.65:58671)
0807 03:01:05.929913 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:05.929976 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:05.932097 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:05.932874 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:05.932888 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:05.933675 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:05.934530 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:05.934557 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:05.935633 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:05.935641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:05.937962 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:05.937982 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:05.939171 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:05.939181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:05.939547 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:05.940320 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:05.940355 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:05.943334 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:05.946852 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:05.946861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:05.946880 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:05.947423 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:05.947832 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:05.947837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:05.947841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:05.948113 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:05.948689 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:05.948698 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:05.949164 (+ 466us) client_negotiation.cc:769] Sending connection context
0807 03:01:05.949441 (+ 277us) client_negotiation.cc:240] Negotiation successful
0807 03:01:05.950034 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":343,"threads_started":1}
I20260807 03:01:05.953034 29306 heartbeater.cc:347] Connected to a master server at 127.1.121.126:36453
I20260807 03:01:05.953375 29306 heartbeater.cc:464] Registering TS with master...
I20260807 03:01:05.954166 29306 heartbeater.cc:511] Master 127.1.121.126:36453 requested a full tablet report, sending...
I20260807 03:01:05.956578 29048 ts_manager.cc:194] Registered new tserver with Master: bbc306f2a23445c39fee26340fef0b69 (127.1.121.65:38911)
I20260807 03:01:05.958128 29048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.65@KRBTEST.COM'} at 127.1.121.65:58671
WARNING: no policy specified for kudu/127.1.121.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
Entry for principal kudu/127.1.121.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab.
I20260807 03:01:06.009208 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:36453
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:06.395502 29316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:06.395821 29316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:06.395934 29316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:06.404421 29316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:06.404546 29316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:06.404660 29316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 03:01:06.415086 29316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.66.keytab
--principal=kudu/127.1.121.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:36453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:06.416414 29316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:06.417730 29316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:06.431396 29321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:06.431439 29322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:06.431471 29324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:06.432665 29316 server_base.cc:1034] running on GCE node
I20260807 03:01:06.433545 29316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:06.434639 29316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:06.435861 29316 hybrid_clock.cc:648] HybridClock initialized: now 1786071666435789 us; error 56 us; skew 500 ppm
I20260807 03:01:06.436281 29316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:06 dist-test-slave-3h5h krb5kdc[28978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071666, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:06.439924 29316 init.cc:377] Logged in from keytab as kudu/127.1.121.66@KRBTEST.COM (short username kudu)
I20260807 03:01:06.441541 29316 webserver.cc:460] Webserver started at http://127.1.121.66:45169/ using document root <none> and password file <none>
I20260807 03:01:06.442284 29316 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:06.442441 29316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:06.442817 29316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:06.446121 29316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "e9fed01398704196bb1f421fbb8ad8bb"
format_stamp: "Formatted at 2026-08-07 03:01:06 on dist-test-slave-3h5h"
server_key: "58f69a87a93375eb582ca4a316c005d8"
server_key_iv: "43b5968f702d9e5867e3ad83263ef782"
server_key_version: "encryptionkey@0"
I20260807 03:01:06.446964 29316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "e9fed01398704196bb1f421fbb8ad8bb"
format_stamp: "Formatted at 2026-08-07 03:01:06 on dist-test-slave-3h5h"
server_key: "58f69a87a93375eb582ca4a316c005d8"
server_key_iv: "43b5968f702d9e5867e3ad83263ef782"
server_key_version: "encryptionkey@0"
I20260807 03:01:06.452680 29316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:01:06.456233 29335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:06.457660 29316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:01:06.457855 29316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "e9fed01398704196bb1f421fbb8ad8bb"
format_stamp: "Formatted at 2026-08-07 03:01:06 on dist-test-slave-3h5h"
server_key: "58f69a87a93375eb582ca4a316c005d8"
server_key_iv: "43b5968f702d9e5867e3ad83263ef782"
server_key_version: "encryptionkey@0"
I20260807 03:01:06.458092 29316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:06.488217 29316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:06.492995 29316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:06.493319 29316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:06.494364 29316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:01:06.495756 29316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:01:06.495862 29316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:06.496013 29316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:01:06.496099 29316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:06.530799 29316 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:36627
I20260807 03:01:06.530843 29491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:36627 every 8 connection(s)
I20260807 03:01:06.532196 29316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 03:01:06.534875 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 29316
I20260807 03:01:06.535073 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 03:01:06.535538 1509 external_mini_cluster.cc:1428] Setting key 72dcb0ad83195fc172068e893cea2ff2
Aug 07 03:01:06 dist-test-slave-3h5h krb5kdc[28978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071666, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.66@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:01:06.555111 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:06.534974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:41851 (local address 127.1.121.126:36453)
0807 03:01:06.538073 (+ 3099us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:06.538079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:06.538100 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:01:06.539094 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:06.539099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:06.539194 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:06.539324 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:06.540887 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:06.541619 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:06.543309 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:06.543815 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:06.543980 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:06.547687 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:06.547696 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:06.547705 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:06.547752 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:06.551545 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:06.552476 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:06.552485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:06.552492 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:06.552647 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:06.553172 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:06.553178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:06.553182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:06.553518 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:06.553712 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:06.554457 (+ 745us) server_negotiation.cc:299] Negotiation successful
0807 03:01:06.554702 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2972,"thread_start_us":2761,"threads_started":1}
I20260807 03:01:06.556139 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:06.535619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:36453 (local address 127.1.121.66:41851)
0807 03:01:06.536597 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:06.536658 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:06.538817 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:06.539590 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:06.539600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:06.540106 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:06.540691 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:06.540710 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:06.541797 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:06.541801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:06.543163 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:06.543176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:06.544076 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:06.544085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:06.544327 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:06.545084 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:06.545107 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:06.547409 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:06.551721 (+ 4312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:06.551733 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:06.551753 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:06.552341 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:06.552774 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:06.552780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:06.552786 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:06.553051 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:06.553667 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:06.553676 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:06.554066 (+ 390us) client_negotiation.cc:769] Sending connection context
0807 03:01:06.554355 (+ 289us) client_negotiation.cc:240] Negotiation successful
0807 03:01:06.554934 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":392,"threads_started":1}
I20260807 03:01:06.557932 29492 heartbeater.cc:347] Connected to a master server at 127.1.121.126:36453
I20260807 03:01:06.558295 29492 heartbeater.cc:464] Registering TS with master...
I20260807 03:01:06.559231 29492 heartbeater.cc:511] Master 127.1.121.126:36453 requested a full tablet report, sending...
I20260807 03:01:06.560997 29048 ts_manager.cc:194] Registered new tserver with Master: e9fed01398704196bb1f421fbb8ad8bb (127.1.121.66:36627)
I20260807 03:01:06.562386 29048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.66@KRBTEST.COM'} at 127.1.121.66:41851
WARNING: no policy specified for kudu/127.1.121.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
Entry for principal kudu/127.1.121.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab.
I20260807 03:01:06.614334 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:36453
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:01:06.961417 29306 heartbeater.cc:503] Master 127.1.121.126:36453 was elected leader, sending a full tablet report...
W20260807 03:01:06.997740 29502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:06.998126 29502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:06.998318 29502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:07.006188 29502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:07.006362 29502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:07.006544 29502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 03:01:07.016775 29502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.67.keytab
--principal=kudu/127.1.121.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:36453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:07.018224 29502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:07.019665 29502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:07.033574 29516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:07.033663 29518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:07.033638 29515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:07.034349 29502 server_base.cc:1034] running on GCE node
I20260807 03:01:07.034986 29502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:07.036037 29502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:07.037287 29502 hybrid_clock.cc:648] HybridClock initialized: now 1786071667037212 us; error 65 us; skew 500 ppm
I20260807 03:01:07.037684 29502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:07 dist-test-slave-3h5h krb5kdc[28978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071667, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:07.041173 29502 init.cc:377] Logged in from keytab as kudu/127.1.121.67@KRBTEST.COM (short username kudu)
I20260807 03:01:07.042760 29502 webserver.cc:460] Webserver started at http://127.1.121.67:44391/ using document root <none> and password file <none>
I20260807 03:01:07.043570 29502 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:07.043722 29502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:07.044070 29502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:07.047256 29502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "bfdaa0d050184cfe85c57e2c22c3807d"
format_stamp: "Formatted at 2026-08-07 03:01:07 on dist-test-slave-3h5h"
server_key: "545349143f3eb460b67e106f54794c28"
server_key_iv: "3acc964065d13a863b4f34767731598c"
server_key_version: "encryptionkey@0"
I20260807 03:01:07.048179 29502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "bfdaa0d050184cfe85c57e2c22c3807d"
format_stamp: "Formatted at 2026-08-07 03:01:07 on dist-test-slave-3h5h"
server_key: "545349143f3eb460b67e106f54794c28"
server_key_iv: "3acc964065d13a863b4f34767731598c"
server_key_version: "encryptionkey@0"
I20260807 03:01:07.053481 29502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260807 03:01:07.057108 29526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:07.058375 29502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:01:07.058554 29502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "bfdaa0d050184cfe85c57e2c22c3807d"
format_stamp: "Formatted at 2026-08-07 03:01:07 on dist-test-slave-3h5h"
server_key: "545349143f3eb460b67e106f54794c28"
server_key_iv: "3acc964065d13a863b4f34767731598c"
server_key_version: "encryptionkey@0"
I20260807 03:01:07.058763 29502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:07.077555 29502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:07.082588 29502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:07.083024 29502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:07.084120 29502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:01:07.085650 29502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:01:07.085763 29502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:07.085930 29502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:01:07.086016 29502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:07.121380 29502 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:38505
I20260807 03:01:07.121443 29685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:38505 every 8 connection(s)
I20260807 03:01:07.122741 29502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 03:01:07.130518 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 29502
I20260807 03:01:07.130714 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 03:01:07.131184 1509 external_mini_cluster.cc:1428] Setting key 7e79633e15149e4a9c543a457e536602
Aug 07 03:01:07 dist-test-slave-3h5h krb5kdc[28978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071667, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.67@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:01:07.143515 29689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.125432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:51107 (local address 127.1.121.126:36453)
0807 03:01:07.125953 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.125958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.126792 (+ 834us) server_negotiation.cc:407] Connection header received
0807 03:01:07.128630 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.128635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.128735 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.128882 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:07.130351 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.131281 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.133179 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.133808 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.133951 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.137540 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:07.137545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:07.137549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:07.137581 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:07.140204 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.141007 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.141013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.141015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.141133 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.141600 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.141604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.141606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.141874 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:07.142036 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.142832 (+ 796us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.143064 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":278,"threads_started":1}
I20260807 03:01:07.144507 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.125803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:36453 (local address 127.1.121.67:51107)
0807 03:01:07.126617 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:07.126658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:07.128356 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:07.129127 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:07.129135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:07.129638 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:07.130182 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.130198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.131447 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.131452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.133001 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.133016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.134069 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.134085 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.134291 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.134981 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:07.135008 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:07.137326 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:07.140351 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:07.140361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:07.140378 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:07.140879 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:07.141260 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:07.141265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:07.141269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:07.141467 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:07.142009 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:07.142018 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:07.142434 (+ 416us) client_negotiation.cc:769] Sending connection context
0807 03:01:07.142722 (+ 288us) client_negotiation.cc:240] Negotiation successful
0807 03:01:07.143305 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":386,"threads_started":1}
I20260807 03:01:07.146087 29686 heartbeater.cc:347] Connected to a master server at 127.1.121.126:36453
I20260807 03:01:07.146456 29686 heartbeater.cc:464] Registering TS with master...
I20260807 03:01:07.147411 29686 heartbeater.cc:511] Master 127.1.121.126:36453 requested a full tablet report, sending...
I20260807 03:01:07.149152 29047 ts_manager.cc:194] Registered new tserver with Master: bfdaa0d050184cfe85c57e2c22c3807d (127.1.121.67:38505)
I20260807 03:01:07.150089 29047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.121.67@KRBTEST.COM'} at 127.1.121.67:51107
I20260807 03:01:07.159725 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:01:07.177449 29689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.164328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.1.121.126:36453)
0807 03:01:07.164556 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.164565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.164987 (+ 422us) server_negotiation.cc:407] Connection header received
0807 03:01:07.165229 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.165234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.165302 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.165411 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:07.166966 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.167710 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.169120 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.169549 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.169703 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.172089 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:07.172095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:07.172098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:07.172133 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:07.174612 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.175337 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.175344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.175347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.175480 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.175876 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.175881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.175884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.176234 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:07.176494 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.176881 (+ 387us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.177139 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:01:07.181533 29047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:01:07.188469 29047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:01:07.216512 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.202135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38911 (local address 127.0.0.1:39308)
0807 03:01:07.203779 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:07.203829 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:07.203986 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:07.205565 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:07.205574 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:07.205610 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:07.206118 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.206134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.207804 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.207812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.211842 (+ 4030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.211857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.215439 (+ 3582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.215447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.215647 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.215713 (+ 66us) client_negotiation.cc:769] Sending connection context
0807 03:01:07.215853 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 03:01:07.216071 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"thread_start_us":5801,"threads_started":1}
I20260807 03:01:07.217379 29704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.202343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:38505 (local address 127.0.0.1:39542)
0807 03:01:07.208377 (+ 6034us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:07.208406 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:07.208629 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:07.209640 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:07.209645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:07.209672 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:07.210153 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.210177 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.212196 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.212201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.214338 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.214356 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.216447 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.216453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.216775 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.216807 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 03:01:07.216917 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 03:01:07.217040 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5894,"thread_start_us":1054,"threads_started":1}
I20260807 03:01:07.217717 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.202733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.1.121.65:38911)
0807 03:01:07.204995 (+ 2262us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.205001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.205025 (+ 24us) server_negotiation.cc:407] Connection header received
0807 03:01:07.205102 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.205108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.205304 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.206345 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:07.206596 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.207654 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.212047 (+ 4393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.214901 (+ 2854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.215042 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.215294 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.217022 (+ 1728us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.217215 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2055,"thread_start_us":1918,"threads_started":1}
I20260807 03:01:07.218185 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.203155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.1.121.67:38505)
0807 03:01:07.204536 (+ 1381us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.204543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.208515 (+ 3972us) server_negotiation.cc:407] Connection header received
0807 03:01:07.208879 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.208887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.209126 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.209321 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:07.210395 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.212004 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.214525 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.215933 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.216025 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.216245 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.217607 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.217864 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":1105,"threads_started":1}
I20260807 03:01:07.223083 29592 tablet_service.cc:1467] Processing CreateTablet for tablet d5389d21e24447008d6191c026e31612 (DEFAULT_TABLE table=test-table [id=e1ef1e62205f44b9b1daa47d6403dd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:01:07.223083 29224 tablet_service.cc:1467] Processing CreateTablet for tablet d5389d21e24447008d6191c026e31612 (DEFAULT_TABLE table=test-table [id=e1ef1e62205f44b9b1daa47d6403dd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:01:07.224751 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5389d21e24447008d6191c026e31612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:07.225121 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.208697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.1.121.66:36627)
0807 03:01:07.219190 (+ 10493us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.219197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.219254 (+ 57us) server_negotiation.cc:407] Connection header received
0807 03:01:07.219376 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.219386 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.219641 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.219883 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:07.220280 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.221476 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.223002 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.224037 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.224205 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.224405 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.224603 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.224778 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10346,"thread_start_us":11414,"threads_started":1}
I20260807 03:01:07.225102 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.210105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:36627 (local address 127.0.0.1:42922)
0807 03:01:07.218123 (+ 8018us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:07.218143 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:07.218339 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:07.219881 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:07.219886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:07.219917 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:07.220090 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.220100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.221628 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.221635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.222832 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.222845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.224216 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.224222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.224391 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.224428 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 03:01:07.224555 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 03:01:07.224713 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7926,"thread_start_us":8417,"threads_started":1}
I20260807 03:01:07.225888 29592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5389d21e24447008d6191c026e31612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:07.228680 29404 tablet_service.cc:1467] Processing CreateTablet for tablet d5389d21e24447008d6191c026e31612 (DEFAULT_TABLE table=test-table [id=e1ef1e62205f44b9b1daa47d6403dd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:01:07.230062 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5389d21e24447008d6191c026e31612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:07.240343 29713 tablet_bootstrap.cc:492] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: Bootstrap starting.
I20260807 03:01:07.242476 29716 tablet_bootstrap.cc:492] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: Bootstrap starting.
I20260807 03:01:07.242614 29717 tablet_bootstrap.cc:492] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: Bootstrap starting.
I20260807 03:01:07.245662 29717 tablet_bootstrap.cc:654] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:07.245700 29716 tablet_bootstrap.cc:654] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:07.246774 29717 log.cc:824] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:07.246951 29716 log.cc:824] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:07.249068 29713 tablet_bootstrap.cc:654] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:07.250005 29713 log.cc:824] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:07.250358 29716 tablet_bootstrap.cc:492] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: No bootstrap required, opened a new log
I20260807 03:01:07.250356 29717 tablet_bootstrap.cc:492] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: No bootstrap required, opened a new log
I20260807 03:01:07.250649 29717 ts_tablet_manager.cc:1397] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 03:01:07.250649 29716 ts_tablet_manager.cc:1397] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 03:01:07.252943 29713 tablet_bootstrap.cc:492] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: No bootstrap required, opened a new log
I20260807 03:01:07.253216 29713 ts_tablet_manager.cc:1397] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20260807 03:01:07.259960 29713 raft_consensus.cc:348] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } }
I20260807 03:01:07.260325 29713 raft_consensus.cc:374] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:07.260432 29713 raft_consensus.cc:729] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc306f2a23445c39fee26340fef0b69, State: Initialized, Role: FOLLOWER
I20260807 03:01:07.261085 29713 consensus_queue.cc:260] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } }
I20260807 03:01:07.261613 29717 raft_consensus.cc:348] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } }
I20260807 03:01:07.261612 29716 raft_consensus.cc:348] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } }
I20260807 03:01:07.262105 29717 raft_consensus.cc:374] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:07.262111 29716 raft_consensus.cc:374] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:07.262214 29717 raft_consensus.cc:729] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fed01398704196bb1f421fbb8ad8bb, State: Initialized, Role: FOLLOWER
I20260807 03:01:07.262218 29716 raft_consensus.cc:729] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfdaa0d050184cfe85c57e2c22c3807d, State: Initialized, Role: FOLLOWER
I20260807 03:01:07.262804 29716 consensus_queue.cc:260] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } }
I20260807 03:01:07.262917 29717 consensus_queue.cc:260] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } }
I20260807 03:01:07.263095 29713 ts_tablet_manager.cc:1428] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260807 03:01:07.264518 29686 heartbeater.cc:503] Master 127.1.121.126:36453 was elected leader, sending a full tablet report...
I20260807 03:01:07.264518 29492 heartbeater.cc:503] Master 127.1.121.126:36453 was elected leader, sending a full tablet report...
I20260807 03:01:07.264652 29716 ts_tablet_manager.cc:1428] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260807 03:01:07.265152 29717 ts_tablet_manager.cc:1428] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
W20260807 03:01:07.284354 29496 tablet.cc:2369] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:01:07.359550 29722 raft_consensus.cc:482] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:01:07.359859 29722 raft_consensus.cc:504] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } }
I20260807 03:01:07.362124 29722 leader_election.cc:288] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbc306f2a23445c39fee26340fef0b69 (127.1.121.65:38911), e9fed01398704196bb1f421fbb8ad8bb (127.1.121.66:36627)
I20260807 03:01:07.369204 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.362463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:59601 (local address 127.1.121.65:38911)
0807 03:01:07.362718 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.362723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.362936 (+ 213us) server_negotiation.cc:407] Connection header received
0807 03:01:07.363224 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.363230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.363314 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.363421 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:07.364016 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.365225 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.367167 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.368334 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.368461 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.368513 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.368920 (+ 407us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.369017 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 03:01:07.369292 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.362576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:38911 (local address 127.1.121.67:59601)
0807 03:01:07.362809 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:07.362833 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:07.363000 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:07.363457 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:07.363462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:07.363492 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:07.363798 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.363820 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.365397 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.365402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.366992 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.367005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.368486 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.368494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.368699 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.368737 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 03:01:07.368862 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 03:01:07.369006 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 03:01:07.370203 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.363284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:36593 (local address 127.1.121.66:36627)
0807 03:01:07.363553 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.363562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.364131 (+ 569us) server_negotiation.cc:407] Connection header received
0807 03:01:07.364265 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.364271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.364364 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.364517 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:07.365069 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.366099 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.368211 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.369416 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.369582 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.369665 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.369911 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.370020 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 03:01:07.370307 29724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.363138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:36627 (local address 127.1.121.67:36593)
0807 03:01:07.364028 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:07.364042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:07.364138 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:07.364537 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:07.364540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:07.364559 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:07.364920 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.364935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.366271 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.366275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.368035 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:07.368052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.369629 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.369634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:07.369753 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.369775 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:01:07.369850 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 03:01:07.369977 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":570,"threads_started":1}
I20260807 03:01:07.370501 29252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5389d21e24447008d6191c026e31612" candidate_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc306f2a23445c39fee26340fef0b69" is_pre_election: true
I20260807 03:01:07.371063 29252 raft_consensus.cc:2454] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfdaa0d050184cfe85c57e2c22c3807d in term 0.
I20260807 03:01:07.371219 29432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5389d21e24447008d6191c026e31612" candidate_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fed01398704196bb1f421fbb8ad8bb" is_pre_election: true
I20260807 03:01:07.371694 29432 raft_consensus.cc:2454] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfdaa0d050184cfe85c57e2c22c3807d in term 0.
I20260807 03:01:07.371737 29531 leader_election.cc:302] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbc306f2a23445c39fee26340fef0b69, bfdaa0d050184cfe85c57e2c22c3807d; no voters:
I20260807 03:01:07.372196 29722 raft_consensus.cc:2781] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:01:07.372375 29722 raft_consensus.cc:482] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:01:07.372498 29722 raft_consensus.cc:3037] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:01:07.374437 29688 tablet.cc:2369] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:01:07.375262 29722 raft_consensus.cc:504] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } }
I20260807 03:01:07.375780 29722 leader_election.cc:288] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [CANDIDATE]: Term 1 election: Requested vote from peers bbc306f2a23445c39fee26340fef0b69 (127.1.121.65:38911), e9fed01398704196bb1f421fbb8ad8bb (127.1.121.66:36627)
I20260807 03:01:07.376266 29252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5389d21e24447008d6191c026e31612" candidate_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc306f2a23445c39fee26340fef0b69"
I20260807 03:01:07.376374 29432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5389d21e24447008d6191c026e31612" candidate_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fed01398704196bb1f421fbb8ad8bb"
I20260807 03:01:07.376471 29252 raft_consensus.cc:3037] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:07.376609 29432 raft_consensus.cc:3037] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:07.378799 29432 raft_consensus.cc:2454] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfdaa0d050184cfe85c57e2c22c3807d in term 1.
I20260807 03:01:07.378801 29252 raft_consensus.cc:2454] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfdaa0d050184cfe85c57e2c22c3807d in term 1.
I20260807 03:01:07.379222 29531 leader_election.cc:302] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbc306f2a23445c39fee26340fef0b69, bfdaa0d050184cfe85c57e2c22c3807d; no voters:
I20260807 03:01:07.379503 29722 raft_consensus.cc:2781] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:07.379902 29722 raft_consensus.cc:686] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 1 LEADER]: Becoming Leader. State: Replica: bfdaa0d050184cfe85c57e2c22c3807d, State: Running, Role: LEADER
I20260807 03:01:07.380512 29722 consensus_queue.cc:237] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } }
I20260807 03:01:07.385243 29043 catalog_manager.cc:5495] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d reported cstate change: term changed from 0 to 1, leader changed from <none> to bfdaa0d050184cfe85c57e2c22c3807d (127.1.121.67). New cstate: current_term: 1 leader_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfdaa0d050184cfe85c57e2c22c3807d" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 } health_report { overall_health: UNKNOWN } } }
I20260807 03:01:07.405493 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.400071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.1.121.67:38505)
0807 03:01:07.400304 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.400310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.400380 (+ 70us) server_negotiation.cc:407] Connection header received
0807 03:01:07.400569 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.400573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.400651 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.400779 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:01:07.401182 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.401925 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.403264 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.403795 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.403893 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.404350 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:01:07.404912 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:01:07.405027 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.405108 (+ 81us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.405188 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:01:07.413676 29432 raft_consensus.cc:1261] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [term 1 FOLLOWER]: Refusing update from remote peer bfdaa0d050184cfe85c57e2c22c3807d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:01:07.413730 29252 raft_consensus.cc:1261] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [term 1 FOLLOWER]: Refusing update from remote peer bfdaa0d050184cfe85c57e2c22c3807d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:01:07.414696 29725 consensus_queue.cc:1035] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc306f2a23445c39fee26340fef0b69" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:01:07.415105 29722 consensus_queue.cc:1035] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fed01398704196bb1f421fbb8ad8bb" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 36627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:01:07.425295 29732 mvcc.cc:205] Tried to move back new op lower bound from 7315749549715927040 to 7315749549594669056. Current Snapshot: MvccSnapshot[applied={T|T < 7315749549715927040}]
W20260807 03:01:07.428231 29307 tablet.cc:2369] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:01:07.442042 29045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:01:07.442363 29045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:01:07.446328 29045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9e04b1ce5914d25985db3f7caa5808b: Sending DeleteTablet for 3 replicas of tablet d5389d21e24447008d6191c026e31612
I20260807 03:01:07.447630 29592 tablet_service.cc:1514] Processing DeleteTablet for tablet d5389d21e24447008d6191c026e31612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:01:07 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:39542
I20260807 03:01:07.447723 29404 tablet_service.cc:1514] Processing DeleteTablet for tablet d5389d21e24447008d6191c026e31612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:01:07 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:42922
I20260807 03:01:07.448889 29224 tablet_service.cc:1514] Processing DeleteTablet for tablet d5389d21e24447008d6191c026e31612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:01:07 UTC) from {username='kudu', principal='kudu/127.1.121.126@KRBTEST.COM'} at 127.0.0.1:39308
I20260807 03:01:07.448992 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 29130
I20260807 03:01:07.458607 29743 tablet_replica.cc:330] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: stopping tablet replica
I20260807 03:01:07.459231 29743 raft_consensus.cc:2229] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:01:07.460067 29743 raft_consensus.cc:2258] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:01:07.462620 29743 ts_tablet_manager.cc:1905] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:01:07.463779 29745 tablet_replica.cc:330] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: stopping tablet replica
I20260807 03:01:07.464208 29745 raft_consensus.cc:2229] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:01:07.464766 29745 raft_consensus.cc:2258] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:01:07.468243 29744 tablet_replica.cc:330] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: stopping tablet replica
I20260807 03:01:07.468672 29744 raft_consensus.cc:2229] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:01:07.469193 29744 raft_consensus.cc:2258] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:01:07.470767 29743 ts_tablet_manager.cc:1918] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:01:07.470952 29743 log.cc:1196] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/d5389d21e24447008d6191c026e31612
I20260807 03:01:07.471535 29743 ts_tablet_manager.cc:1939] T d5389d21e24447008d6191c026e31612 P bfdaa0d050184cfe85c57e2c22c3807d: Deleting consensus metadata
I20260807 03:01:07.471572 29745 ts_tablet_manager.cc:1905] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:01:07.473459 29030 catalog_manager.cc:4841] TS bfdaa0d050184cfe85c57e2c22c3807d (127.1.121.67:38505): tablet d5389d21e24447008d6191c026e31612 (table test-table [id=e1ef1e62205f44b9b1daa47d6403dd74]) successfully deleted
Aug 07 03:01:07 dist-test-slave-3h5h krb5kdc[28978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.65@KRBTEST.COM
I20260807 03:01:07.477305 29744 ts_tablet_manager.cc:1905] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:01:07.478579 29745 ts_tablet_manager.cc:1918] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:01:07.478755 29745 log.cc:1196] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/d5389d21e24447008d6191c026e31612
I20260807 03:01:07.479324 29745 ts_tablet_manager.cc:1939] T d5389d21e24447008d6191c026e31612 P bbc306f2a23445c39fee26340fef0b69: Deleting consensus metadata
I20260807 03:01:07.481510 29029 catalog_manager.cc:4841] TS bbc306f2a23445c39fee26340fef0b69 (127.1.121.65:38911): tablet d5389d21e24447008d6191c026e31612 (table test-table [id=e1ef1e62205f44b9b1daa47d6403dd74]) successfully deleted
I20260807 03:01:07.483290 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.450155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.1.121.65:38911)
0807 03:01:07.454715 (+ 4560us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.454722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.460410 (+ 5688us) server_negotiation.cc:407] Connection header received
0807 03:01:07.460962 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.460967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.461084 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.461169 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:07.464958 (+ 3789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.466079 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.467960 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.468649 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.468784 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.475211 (+ 6427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:07.475232 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:07.475253 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:07.475293 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:07.479627 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.480547 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.480556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.480560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.480749 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.481273 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.481279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.481282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.481750 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:07.482032 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.482565 (+ 533us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.482793 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4420}
I20260807 03:01:07.484020 29300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:07.485263 29744 ts_tablet_manager.cc:1918] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:01:07.485481 29744 log.cc:1196] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/d5389d21e24447008d6191c026e31612
I20260807 03:01:07.486402 29744 ts_tablet_manager.cc:1939] T d5389d21e24447008d6191c026e31612 P e9fed01398704196bb1f421fbb8ad8bb: Deleting consensus metadata
I20260807 03:01:07.489024 29029 catalog_manager.cc:4841] TS e9fed01398704196bb1f421fbb8ad8bb (127.1.121.66:36627): tablet d5389d21e24447008d6191c026e31612 (table test-table [id=e1ef1e62205f44b9b1daa47d6403dd74]) successfully deleted
I20260807 03:01:07.576228 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 29130
I20260807 03:01:07.606389 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 29316
Aug 07 03:01:07 dist-test-slave-3h5h krb5kdc[28978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.66@KRBTEST.COM
I20260807 03:01:07.627988 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.607363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.1.121.66:36627)
0807 03:01:07.607534 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.607539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.607556 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:01:07.607637 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.607640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.607717 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.607801 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:07.609566 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.610327 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.611631 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.612026 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.612166 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.620726 (+ 8560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:07.620752 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:07.620777 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:07.620988 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:07.624901 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.625750 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.625761 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.625765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.625934 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.626322 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.626327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.626331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.626844 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:07.627163 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.627349 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.627585 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:01:07.628968 29484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:07.720183 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 29316
I20260807 03:01:07.746354 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 29502
Aug 07 03:01:07 dist-test-slave-3h5h krb5kdc[28978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.67@KRBTEST.COM
I20260807 03:01:07.766503 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:07.747369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.1.121.67:38505)
0807 03:01:07.747569 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:07.747573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:07.747592 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:01:07.747763 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:07.747767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:07.747858 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:07.747982 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:07.749862 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.750747 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.752704 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:07.753231 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:07.753363 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:07.759346 (+ 5983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:07.759362 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:07.759383 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:07.759425 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:07.763332 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.764112 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.764124 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.764128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.764346 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:07.764723 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:07.764729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:07.764732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:07.765299 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:07.765637 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:07.765803 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 03:01:07.766055 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:01:07.767282 29680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:07.897303 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 29502
I20260807 03:01:07.926292 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 29003
I20260807 03:01:07.927500 29088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:08.042495 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 29003
2026-08-07T03:01:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5085 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T03:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:08Z Disabled control of system clock
I20260807 03:01:08.150384 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34217
--webserver_interface=127.1.121.126
--webserver_port=0
--builtin_ntp_servers=127.1.121.84:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:34217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:01:08.524096 29761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:08.524403 29761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:08.524489 29761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:08.532030 29761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:08.532174 29761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:08.532218 29761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:08.532279 29761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:08.532325 29761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:08.541870 29761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:34217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:08.543258 29761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:08.545123 29761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:08.557354 29770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:08.557453 29771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:08.557488 29774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:08.558332 29761 server_base.cc:1034] running on GCE node
I20260807 03:01:08.559121 29761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:08.560555 29761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:08.561790 29761 hybrid_clock.cc:648] HybridClock initialized: now 1786071668561706 us; error 73 us; skew 500 ppm
I20260807 03:01:08.562275 29761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:01:08.565223 29761 webserver.cc:460] Webserver started at http://127.1.121.126:41883/ using document root <none> and password file <none>
I20260807 03:01:08.566045 29761 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:08.566157 29761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:08.566587 29761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:08.569933 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "289934dd78a040d2bb53368d59c0d5cf"
format_stamp: "Formatted at 2026-08-07 03:01:08 on dist-test-slave-3h5h"
server_key: "0cdb6e08687a10772be022001f3f8b6f"
server_key_iv: "7d8929fc489cb5092b9a731786bebb61"
server_key_version: "encryptionkey@0"
I20260807 03:01:08.570973 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "289934dd78a040d2bb53368d59c0d5cf"
format_stamp: "Formatted at 2026-08-07 03:01:08 on dist-test-slave-3h5h"
server_key: "0cdb6e08687a10772be022001f3f8b6f"
server_key_iv: "7d8929fc489cb5092b9a731786bebb61"
server_key_version: "encryptionkey@0"
I20260807 03:01:08.576907 29761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 03:01:08.580914 29786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:08.582561 29761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:01:08.582788 29761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "289934dd78a040d2bb53368d59c0d5cf"
format_stamp: "Formatted at 2026-08-07 03:01:08 on dist-test-slave-3h5h"
server_key: "0cdb6e08687a10772be022001f3f8b6f"
server_key_iv: "7d8929fc489cb5092b9a731786bebb61"
server_key_version: "encryptionkey@0"
I20260807 03:01:08.583101 29761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:08.604537 29761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:08.605559 29761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:08.605887 29761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:08.626497 29761 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:34217
I20260807 03:01:08.626539 29857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:34217 every 8 connection(s)
I20260807 03:01:08.628041 29761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:08.632354 29858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:08.634557 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 29761
I20260807 03:01:08.634792 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:08.635262 1509 external_mini_cluster.cc:1428] Setting key 26f1442242503a5d01ca082a3515a145
I20260807 03:01:08.646718 29858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf: Bootstrap starting.
I20260807 03:01:08.650872 29858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:08.651764 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:08.637468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.1.121.126:34217)
0807 03:01:08.638624 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:08.638638 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:08.638692 (+ 54us) server_negotiation.cc:407] Connection header received
0807 03:01:08.641127 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:08.641158 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:08.641759 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:08.642216 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:08.642799 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:08.644955 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:08.646326 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:08.647369 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:08.647553 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:08.648743 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:08.648754 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:08.648774 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:08.649565 (+ 791us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:08.649690 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:08.649718 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:08.649849 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:08.650071 (+ 222us) server_negotiation.cc:299] Negotiation successful
0807 03:01:08.650391 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":346,"threads_started":1}
I20260807 03:01:08.652266 29858 log.cc:824] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:08.655642 29858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf: No bootstrap required, opened a new log
I20260807 03:01:08.664077 29858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289934dd78a040d2bb53368d59c0d5cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34217 } }
I20260807 03:01:08.664534 29858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:08.664661 29858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289934dd78a040d2bb53368d59c0d5cf, State: Initialized, Role: FOLLOWER
I20260807 03:01:08.665448 29858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289934dd78a040d2bb53368d59c0d5cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34217 } }
I20260807 03:01:08.665791 29858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:08.666000 29858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:08.666210 29858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:08.668429 29858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289934dd78a040d2bb53368d59c0d5cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34217 } }
I20260807 03:01:08.669046 29858 leader_election.cc:302] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 289934dd78a040d2bb53368d59c0d5cf; no voters:
I20260807 03:01:08.669642 29858 leader_election.cc:288] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:08.669826 29863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:08.670579 29863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [term 1 LEADER]: Becoming Leader. State: Replica: 289934dd78a040d2bb53368d59c0d5cf, State: Running, Role: LEADER
I20260807 03:01:08.671095 29863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289934dd78a040d2bb53368d59c0d5cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34217 } }
I20260807 03:01:08.672070 29858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:08.674261 29869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 289934dd78a040d2bb53368d59c0d5cf. Latest consensus state: current_term: 1 leader_uuid: "289934dd78a040d2bb53368d59c0d5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289934dd78a040d2bb53368d59c0d5cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34217 } } }
I20260807 03:01:08.674358 29864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "289934dd78a040d2bb53368d59c0d5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289934dd78a040d2bb53368d59c0d5cf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34217 } } }
I20260807 03:01:08.674671 29869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:08.674757 29864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:08.677675 29877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:08.683018 29877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:08.694977 29877 catalog_manager.cc:1347] Generated new cluster ID: b4f393eb5c5046e7b1e5b96f147f666e
I20260807 03:01:08.695166 29877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:08.725076 29877 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:08.726464 29877 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:08.739253 29877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf: Generated new TSK 0
I20260807 03:01:08.740135 29877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:08.744604 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.65:0
--local_ip_for_outbound_sockets=127.1.121.65
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34217
--builtin_ntp_servers=127.1.121.84:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:01:09.127640 29892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:09.127947 29892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:09.128139 29892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:09.136448 29892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:09.136560 29892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:09.136771 29892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.65
I20260807 03:01:09.149724 29892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.65
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.65
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:09.151597 29892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:09.153631 29892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:09.168332 29898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:09.168531 29899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:09.168604 29901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:09.169898 29892 server_base.cc:1034] running on GCE node
I20260807 03:01:09.170560 29892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:09.171564 29892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:09.172804 29892 hybrid_clock.cc:648] HybridClock initialized: now 1786071669172750 us; error 68 us; skew 500 ppm
I20260807 03:01:09.173267 29892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:01:09.180637 29892 webserver.cc:460] Webserver started at http://127.1.121.65:41439/ using document root <none> and password file <none>
I20260807 03:01:09.181478 29892 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:09.181615 29892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:09.182013 29892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:09.185384 29892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/data/instance:
uuid: "cf3e070fee0e43f597a8d5c466f3a095"
format_stamp: "Formatted at 2026-08-07 03:01:09 on dist-test-slave-3h5h"
server_key: "2355eed1ac9880e797c742220f4a07a0"
server_key_iv: "fe275e5cdf96c32ae856b3d0cbc09b88"
server_key_version: "encryptionkey@0"
I20260807 03:01:09.186218 29892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance:
uuid: "cf3e070fee0e43f597a8d5c466f3a095"
format_stamp: "Formatted at 2026-08-07 03:01:09 on dist-test-slave-3h5h"
server_key: "2355eed1ac9880e797c742220f4a07a0"
server_key_iv: "fe275e5cdf96c32ae856b3d0cbc09b88"
server_key_version: "encryptionkey@0"
I20260807 03:01:09.191675 29892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:01:09.195849 29913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:09.198164 29892 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 03:01:09.198349 29892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/wal
uuid: "cf3e070fee0e43f597a8d5c466f3a095"
format_stamp: "Formatted at 2026-08-07 03:01:09 on dist-test-slave-3h5h"
server_key: "2355eed1ac9880e797c742220f4a07a0"
server_key_iv: "fe275e5cdf96c32ae856b3d0cbc09b88"
server_key_version: "encryptionkey@0"
I20260807 03:01:09.198731 29892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:09.217979 29892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:09.219098 29892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:09.219475 29892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:09.220798 29892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:01:09.223062 29892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:01:09.223186 29892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:09.223321 29892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:01:09.223404 29892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:09.264498 29892 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.65:37197
I20260807 03:01:09.264542 30066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.65:37197 every 8 connection(s)
I20260807 03:01:09.265954 29892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/data/info.pb
I20260807 03:01:09.275859 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 29892
I20260807 03:01:09.276079 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/wal/instance
I20260807 03:01:09.276705 1509 external_mini_cluster.cc:1428] Setting key 097fc4fb86b2aacdbded680825602d8a
I20260807 03:01:09.279203 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:09.268604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:53941 (local address 127.1.121.126:34217)
0807 03:01:09.269162 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:09.269169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:09.270290 (+ 1121us) server_negotiation.cc:407] Connection header received
0807 03:01:09.271993 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:09.271997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:09.272074 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:09.272218 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:09.273424 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:09.274365 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:09.275904 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:09.276676 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:09.276871 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:09.278301 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:09.278306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:09.278310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:09.278328 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:09.278372 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:09.278385 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:09.278465 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:09.278755 (+ 290us) server_negotiation.cc:299] Negotiation successful
0807 03:01:09.278852 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":265,"threads_started":1}
I20260807 03:01:09.279899 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.66:0
--local_ip_for_outbound_sockets=127.1.121.66
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34217
--builtin_ntp_servers=127.1.121.84:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:01:09.280256 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:09.269250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:34217 (local address 127.1.121.65:53941)
0807 03:01:09.270135 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:09.270171 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:09.271762 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:09.272433 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:09.272441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:09.272524 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:09.273204 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:09.273222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:09.274517 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:09.274520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:09.275743 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:09.275755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:09.276958 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:09.276964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:09.277175 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:09.277937 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:01:09.277940 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:09.277994 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:01:09.278091 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:09.278493 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:09.278504 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:09.278512 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 03:01:09.278722 (+ 210us) client_negotiation.cc:240] Negotiation successful
0807 03:01:09.279107 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
I20260807 03:01:09.281942 30068 heartbeater.cc:347] Connected to a master server at 127.1.121.126:34217
I20260807 03:01:09.282251 30068 heartbeater.cc:464] Registering TS with master...
I20260807 03:01:09.282982 30068 heartbeater.cc:511] Master 127.1.121.126:34217 requested a full tablet report, sending...
I20260807 03:01:09.285549 29814 ts_manager.cc:194] Registered new tserver with Master: cf3e070fee0e43f597a8d5c466f3a095 (127.1.121.65:37197)
I20260807 03:01:09.287813 29814 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.65:53941
W20260807 03:01:09.642428 30073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:09.642757 30073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:09.642930 30073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:09.650321 30073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:09.650470 30073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:09.650695 30073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.66
I20260807 03:01:09.659870 30073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.66
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.66
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:09.661335 30073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:09.663177 30073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:09.676769 30079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:09.676815 30083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:09.676769 30080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:09.677647 30073 server_base.cc:1034] running on GCE node
I20260807 03:01:09.678289 30073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:09.679246 30073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:09.680487 30073 hybrid_clock.cc:648] HybridClock initialized: now 1786071669680446 us; error 51 us; skew 500 ppm
I20260807 03:01:09.680882 30073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:01:09.683183 30073 webserver.cc:460] Webserver started at http://127.1.121.66:37149/ using document root <none> and password file <none>
I20260807 03:01:09.683961 30073 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:09.684100 30073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:09.684463 30073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:09.687760 30073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/data/instance:
uuid: "4e2680c59f224624b4af07e1058646c1"
format_stamp: "Formatted at 2026-08-07 03:01:09 on dist-test-slave-3h5h"
server_key: "ef1cd79a2af1edc0c7183e9b4369832b"
server_key_iv: "3abb9ab836a5a2f0c811b46cdd3cfe18"
server_key_version: "encryptionkey@0"
I20260807 03:01:09.688617 30073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance:
uuid: "4e2680c59f224624b4af07e1058646c1"
format_stamp: "Formatted at 2026-08-07 03:01:09 on dist-test-slave-3h5h"
server_key: "ef1cd79a2af1edc0c7183e9b4369832b"
server_key_iv: "3abb9ab836a5a2f0c811b46cdd3cfe18"
server_key_version: "encryptionkey@0"
I20260807 03:01:09.693846 30073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:01:09.697398 30091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:09.698688 30073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:01:09.698819 30073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/wal
uuid: "4e2680c59f224624b4af07e1058646c1"
format_stamp: "Formatted at 2026-08-07 03:01:09 on dist-test-slave-3h5h"
server_key: "ef1cd79a2af1edc0c7183e9b4369832b"
server_key_iv: "3abb9ab836a5a2f0c811b46cdd3cfe18"
server_key_version: "encryptionkey@0"
I20260807 03:01:09.698993 30073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:09.720402 30073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:09.721292 30073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:09.721558 30073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:09.722499 30073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:01:09.723990 30073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:01:09.724080 30073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:09.724171 30073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:01:09.724227 30073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:09.759205 30073 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.66:38829
I20260807 03:01:09.759306 30253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.66:38829 every 8 connection(s)
I20260807 03:01:09.760579 30073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/data/info.pb
I20260807 03:01:09.767172 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 30073
I20260807 03:01:09.767383 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/wal/instance
I20260807 03:01:09.767956 1509 external_mini_cluster.cc:1428] Setting key c536fdb000dbc7eaed3214b16943a901
I20260807 03:01:09.770947 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.67:0
--local_ip_for_outbound_sockets=127.1.121.67
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34217
--builtin_ntp_servers=127.1.121.84:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:01:09.774264 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:09.763350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.66:37169 (local address 127.1.121.126:34217)
0807 03:01:09.763540 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:09.763545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:09.764935 (+ 1390us) server_negotiation.cc:407] Connection header received
0807 03:01:09.766834 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:09.766838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:09.766906 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:09.766995 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:09.768427 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:09.769342 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:09.771198 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:09.771604 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:09.771689 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:09.773243 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:09.773252 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:09.773260 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:09.773286 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:09.773329 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:09.773347 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:09.773445 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:09.773735 (+ 290us) server_negotiation.cc:299] Negotiation successful
0807 03:01:09.773850 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:01:09.775131 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:09.763849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:34217 (local address 127.1.121.66:37169)
0807 03:01:09.764737 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:09.764773 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:09.766480 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:09.767285 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:09.767297 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:09.767421 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:09.768243 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:09.768271 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:09.769500 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:09.769506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:09.771003 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:09.771016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:09.771949 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:09.771953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:09.772148 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:09.772921 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:01:09.772924 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:09.772965 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:01:09.773046 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:09.773451 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:09.773459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:09.773465 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 03:01:09.773720 (+ 255us) client_negotiation.cc:240] Negotiation successful
0807 03:01:09.774017 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":279,"threads_started":1}
I20260807 03:01:09.776628 30254 heartbeater.cc:347] Connected to a master server at 127.1.121.126:34217
I20260807 03:01:09.777004 30254 heartbeater.cc:464] Registering TS with master...
I20260807 03:01:09.777813 30254 heartbeater.cc:511] Master 127.1.121.126:34217 requested a full tablet report, sending...
I20260807 03:01:09.779350 29814 ts_manager.cc:194] Registered new tserver with Master: 4e2680c59f224624b4af07e1058646c1 (127.1.121.66:38829)
I20260807 03:01:09.780181 29814 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.66:37169
W20260807 03:01:10.139580 30260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:10.139950 30260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:10.140136 30260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:10.147578 30260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:10.147745 30260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:10.147925 30260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.67
I20260807 03:01:10.157912 30260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.67
--webserver_port=0
--tserver_master_addrs=127.1.121.126:34217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.67
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:10.159399 30260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:10.161358 30260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:10.175434 30266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:10.175380 30269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:10.175544 30267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:10.175783 30260 server_base.cc:1034] running on GCE node
I20260807 03:01:10.176606 30260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:10.177672 30260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:10.178917 30260 hybrid_clock.cc:648] HybridClock initialized: now 1786071670178859 us; error 46 us; skew 500 ppm
I20260807 03:01:10.179294 30260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:01:10.181783 30260 webserver.cc:460] Webserver started at http://127.1.121.67:42035/ using document root <none> and password file <none>
I20260807 03:01:10.182514 30260 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:10.182655 30260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:10.183040 30260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:10.187784 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/data/instance:
uuid: "8c27dbf67a1f4d699532bc3d457baffe"
format_stamp: "Formatted at 2026-08-07 03:01:10 on dist-test-slave-3h5h"
server_key: "e0d5c8a4e1572d279201a2815af06b06"
server_key_iv: "66c185259bdc69c80772a7485d6b953a"
server_key_version: "encryptionkey@0"
I20260807 03:01:10.188961 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance:
uuid: "8c27dbf67a1f4d699532bc3d457baffe"
format_stamp: "Formatted at 2026-08-07 03:01:10 on dist-test-slave-3h5h"
server_key: "e0d5c8a4e1572d279201a2815af06b06"
server_key_iv: "66c185259bdc69c80772a7485d6b953a"
server_key_version: "encryptionkey@0"
I20260807 03:01:10.196552 30260 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260807 03:01:10.200248 30278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:10.201761 30260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:01:10.201973 30260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/wal
uuid: "8c27dbf67a1f4d699532bc3d457baffe"
format_stamp: "Formatted at 2026-08-07 03:01:10 on dist-test-slave-3h5h"
server_key: "e0d5c8a4e1572d279201a2815af06b06"
server_key_iv: "66c185259bdc69c80772a7485d6b953a"
server_key_version: "encryptionkey@0"
I20260807 03:01:10.202279 30260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:10.216769 30260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:10.217759 30260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:10.218122 30260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:10.219162 30260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:01:10.220747 30260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:01:10.220914 30260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:10.221064 30260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:01:10.221166 30260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:10.257328 30260 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.67:40883
I20260807 03:01:10.257372 30430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.67:40883 every 8 connection(s)
I20260807 03:01:10.258785 30260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/data/info.pb
I20260807 03:01:10.268353 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 30260
I20260807 03:01:10.268540 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/wal/instance
I20260807 03:01:10.269127 1509 external_mini_cluster.cc:1428] Setting key caffe28ecb7d070db82b88ab70da412c
I20260807 03:01:10.272265 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.261467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.67:34933 (local address 127.1.121.126:34217)
0807 03:01:10.261684 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.261688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.263058 (+ 1370us) server_negotiation.cc:407] Connection header received
0807 03:01:10.264895 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.264903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.265001 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.265131 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:10.266159 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.267295 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.269073 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.269555 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.269729 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.271398 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:10.271404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:10.271409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:10.271429 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:10.271468 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:10.271487 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:10.271585 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.271805 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.271906 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 03:01:10.273370 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.261976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:34217 (local address 127.1.121.67:34933)
0807 03:01:10.262904 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.262945 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:10.264586 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:10.265336 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:10.265345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:10.265428 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:10.265979 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.265995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.267446 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.267449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.268929 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.268942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.269852 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.269862 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.270179 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.271019 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:01:10.271023 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:10.271074 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:01:10.271206 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:10.271589 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:10.271597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:10.271603 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 03:01:10.271740 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 03:01:10.272185 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":336,"threads_started":1}
I20260807 03:01:10.274653 30432 heartbeater.cc:347] Connected to a master server at 127.1.121.126:34217
I20260807 03:01:10.274960 30432 heartbeater.cc:464] Registering TS with master...
I20260807 03:01:10.275811 30432 heartbeater.cc:511] Master 127.1.121.126:34217 requested a full tablet report, sending...
I20260807 03:01:10.277300 29814 ts_manager.cc:194] Registered new tserver with Master: 8c27dbf67a1f4d699532bc3d457baffe (127.1.121.67:40883)
I20260807 03:01:10.278113 29814 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.67:34933
I20260807 03:01:10.284754 1509 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:01:10.290951 30068 heartbeater.cc:503] Master 127.1.121.126:34217 was elected leader, sending a full tablet report...
I20260807 03:01:10.295527 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.289216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.1.121.126:34217)
0807 03:01:10.289426 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.289432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.289962 (+ 530us) server_negotiation.cc:407] Connection header received
0807 03:01:10.290102 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.290105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.290171 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.290256 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:10.290726 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.291570 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.293096 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.293530 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.293679 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.294790 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:10.294795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:10.294800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:10.294826 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:10.294859 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:10.294870 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:10.294967 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.295119 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.295210 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 03:01:10.299459 29814 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:01:10.306744 29814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:01:10.332158 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.319036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:40883 (local address 127.0.0.1:56198)
0807 03:01:10.320563 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.320623 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:10.320866 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:10.322100 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:10.322111 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:10.322151 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:10.322592 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.322607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.325028 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.325037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.328066 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.328088 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.330893 (+ 2805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.330902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.331189 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.331262 (+ 73us) client_negotiation.cc:769] Sending connection context
0807 03:01:10.331411 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 03:01:10.331660 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":981,"threads_started":1}
I20260807 03:01:10.333336 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.319034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.1.121.67:40883)
0807 03:01:10.319831 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.319839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.321084 (+ 1245us) server_negotiation.cc:407] Connection header received
0807 03:01:10.321198 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.321208 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.321496 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.321717 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:10.322867 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.324807 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.328347 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.330219 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.330378 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.330740 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.332626 (+ 1886us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.332896 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":482,"threads_started":1}
I20260807 03:01:10.338795 30341 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe7642172964a28b092e8786b1cbf95 (DEFAULT_TABLE table=test-table [id=112e150bda144b6b9683397d6281b701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:01:10.340278 30341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe7642172964a28b092e8786b1cbf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:10.343935 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.319110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:38829 (local address 127.0.0.1:36650)
0807 03:01:10.333717 (+ 14607us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.333742 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:10.333864 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:10.337802 (+ 3938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:10.337810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:10.337838 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:10.338051 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.338061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.339990 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.339995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.341424 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.341437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.343114 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.343120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.343308 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.343345 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 03:01:10.343416 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 03:01:10.343549 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13213,"mutex_wait_us":40,"thread_start_us":18584,"threads_started":1}
I20260807 03:01:10.344511 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.320825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37197 (local address 127.0.0.1:53374)
0807 03:01:10.334688 (+ 13863us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.334710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:10.334837 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:10.336206 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:10.336214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:10.336251 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:10.337529 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.337546 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.339593 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.339598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.341815 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.341832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.343720 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.343727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.343911 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.343951 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 03:01:10.344029 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 03:01:10.344141 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13680,"mutex_wait_us":26,"thread_start_us":13472,"threads_started":1}
I20260807 03:01:10.345153 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.319838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36650 (local address 127.1.121.66:38829)
0807 03:01:10.337204 (+ 17366us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.337213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.337242 (+ 29us) server_negotiation.cc:407] Connection header received
0807 03:01:10.337344 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.337351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.337578 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.338236 (+ 658us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:10.338458 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.339830 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.341578 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.342918 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.344207 (+ 1289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.344428 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.344501 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.344703 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17199,"thread_start_us":17064,"threads_started":1}
I20260807 03:01:10.345408 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.321105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.1.121.65:37197)
0807 03:01:10.335396 (+ 14291us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.335408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.335450 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:01:10.335575 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.335587 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.335923 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.336380 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:10.337771 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.339448 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.342027 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.343514 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.343651 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.343925 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.344897 (+ 972us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.345064 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14112,"thread_start_us":13992,"threads_started":1}
I20260807 03:01:10.348716 29978 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe7642172964a28b092e8786b1cbf95 (DEFAULT_TABLE table=test-table [id=112e150bda144b6b9683397d6281b701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:01:10.349097 30168 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe7642172964a28b092e8786b1cbf95 (DEFAULT_TABLE table=test-table [id=112e150bda144b6b9683397d6281b701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:01:10.350253 29978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe7642172964a28b092e8786b1cbf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:10.350438 30168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe7642172964a28b092e8786b1cbf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:10.355497 30458 tablet_bootstrap.cc:492] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: Bootstrap starting.
I20260807 03:01:10.359548 30458 tablet_bootstrap.cc:654] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:10.360899 30458 log.cc:824] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:10.363735 30460 tablet_bootstrap.cc:492] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: Bootstrap starting.
I20260807 03:01:10.364949 30461 tablet_bootstrap.cc:492] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: Bootstrap starting.
I20260807 03:01:10.365288 30458 tablet_bootstrap.cc:492] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: No bootstrap required, opened a new log
I20260807 03:01:10.365577 30458 ts_tablet_manager.cc:1397] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260807 03:01:10.367326 30460 tablet_bootstrap.cc:654] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:10.368348 30460 log.cc:824] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:10.368445 30461 tablet_bootstrap.cc:654] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:10.369656 30461 log.cc:824] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:10.373204 30460 tablet_bootstrap.cc:492] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: No bootstrap required, opened a new log
I20260807 03:01:10.373204 30461 tablet_bootstrap.cc:492] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: No bootstrap required, opened a new log
I20260807 03:01:10.373482 30460 ts_tablet_manager.cc:1397] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 03:01:10.373482 30461 ts_tablet_manager.cc:1397] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260807 03:01:10.375211 30458 raft_consensus.cc:348] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } }
I20260807 03:01:10.375602 30458 raft_consensus.cc:374] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:10.375710 30458 raft_consensus.cc:729] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c27dbf67a1f4d699532bc3d457baffe, State: Initialized, Role: FOLLOWER
I20260807 03:01:10.376363 30458 consensus_queue.cc:260] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } }
I20260807 03:01:10.377472 30432 heartbeater.cc:503] Master 127.1.121.126:34217 was elected leader, sending a full tablet report...
I20260807 03:01:10.379469 30458 ts_tablet_manager.cc:1428] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260807 03:01:10.384534 30461 raft_consensus.cc:348] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } }
I20260807 03:01:10.384536 30460 raft_consensus.cc:348] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } }
I20260807 03:01:10.385043 30461 raft_consensus.cc:374] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:10.385043 30460 raft_consensus.cc:374] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:10.385152 30461 raft_consensus.cc:729] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2680c59f224624b4af07e1058646c1, State: Initialized, Role: FOLLOWER
I20260807 03:01:10.385167 30460 raft_consensus.cc:729] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3e070fee0e43f597a8d5c466f3a095, State: Initialized, Role: FOLLOWER
I20260807 03:01:10.385826 30460 consensus_queue.cc:260] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } }
I20260807 03:01:10.385823 30461 consensus_queue.cc:260] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } }
I20260807 03:01:10.387921 30460 ts_tablet_manager.cc:1428] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260807 03:01:10.388406 30461 ts_tablet_manager.cc:1428] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260807 03:01:10.388440 30254 heartbeater.cc:503] Master 127.1.121.126:34217 was elected leader, sending a full tablet report...
I20260807 03:01:10.478365 30468 raft_consensus.cc:482] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:01:10.478703 30468 raft_consensus.cc:504] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } }
I20260807 03:01:10.480535 30468 leader_election.cc:288] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c27dbf67a1f4d699532bc3d457baffe (127.1.121.67:40883), 4e2680c59f224624b4af07e1058646c1 (127.1.121.66:38829)
I20260807 03:01:10.488377 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.481067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:41065 (local address 127.1.121.67:40883)
0807 03:01:10.481280 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.481288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.482070 (+ 782us) server_negotiation.cc:407] Connection header received
0807 03:01:10.482276 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.482281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.482370 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.482458 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:10.483175 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.484002 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.486352 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.487312 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.487464 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.487544 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.487966 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.488110 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 03:01:10.488399 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.481123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.121.65:40767 (local address 127.1.121.66:38829)
0807 03:01:10.481364 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.481374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.482263 (+ 889us) server_negotiation.cc:407] Connection header received
0807 03:01:10.482416 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.482420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.482538 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.482641 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:01:10.483357 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.484136 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.486328 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.487442 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.487595 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.487665 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.487975 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.488117 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 03:01:10.488547 30469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.481001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.67:40883 (local address 127.1.121.65:41065)
0807 03:01:10.481925 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.481950 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:10.482114 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:10.482502 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:10.482507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:10.482532 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:10.482985 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.482995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.484154 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.484159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.486059 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.486072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.487480 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.487490 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.487657 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.487815 (+ 158us) client_negotiation.cc:769] Sending connection context
0807 03:01:10.487922 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 03:01:10.488208 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":130,"thread_start_us":468,"threads_started":1}
I20260807 03:01:10.488966 30470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.481218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.66:38829 (local address 127.1.121.65:40767)
0807 03:01:10.482110 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.482127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:10.482283 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:10.482675 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:10.482679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:10.482713 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:01:10.483162 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.483175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.484277 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.484280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.486083 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:10.486095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.487622 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.487627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:10.487778 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.487815 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 03:01:10.487919 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 03:01:10.488168 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":226,"thread_start_us":559,"threads_started":1}
I20260807 03:01:10.489784 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe7642172964a28b092e8786b1cbf95" candidate_uuid: "cf3e070fee0e43f597a8d5c466f3a095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c27dbf67a1f4d699532bc3d457baffe" is_pre_election: true
I20260807 03:01:10.489967 30190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe7642172964a28b092e8786b1cbf95" candidate_uuid: "cf3e070fee0e43f597a8d5c466f3a095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2680c59f224624b4af07e1058646c1" is_pre_election: true
I20260807 03:01:10.490370 30366 raft_consensus.cc:2454] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf3e070fee0e43f597a8d5c466f3a095 in term 0.
I20260807 03:01:10.490672 30190 raft_consensus.cc:2454] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf3e070fee0e43f597a8d5c466f3a095 in term 0.
I20260807 03:01:10.491127 29915 leader_election.cc:302] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c27dbf67a1f4d699532bc3d457baffe, cf3e070fee0e43f597a8d5c466f3a095; no voters:
I20260807 03:01:10.491595 30468 raft_consensus.cc:2781] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:01:10.491755 30468 raft_consensus.cc:482] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:01:10.491861 30468 raft_consensus.cc:3037] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:10.494132 30468 raft_consensus.cc:504] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } }
I20260807 03:01:10.494743 30468 leader_election.cc:288] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [CANDIDATE]: Term 1 election: Requested vote from peers 8c27dbf67a1f4d699532bc3d457baffe (127.1.121.67:40883), 4e2680c59f224624b4af07e1058646c1 (127.1.121.66:38829)
I20260807 03:01:10.495182 30366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe7642172964a28b092e8786b1cbf95" candidate_uuid: "cf3e070fee0e43f597a8d5c466f3a095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c27dbf67a1f4d699532bc3d457baffe"
I20260807 03:01:10.495299 30190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe7642172964a28b092e8786b1cbf95" candidate_uuid: "cf3e070fee0e43f597a8d5c466f3a095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2680c59f224624b4af07e1058646c1"
I20260807 03:01:10.495393 30366 raft_consensus.cc:3037] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:10.495533 30190 raft_consensus.cc:3037] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:10.497920 30366 raft_consensus.cc:2454] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf3e070fee0e43f597a8d5c466f3a095 in term 1.
I20260807 03:01:10.497953 30190 raft_consensus.cc:2454] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf3e070fee0e43f597a8d5c466f3a095 in term 1.
I20260807 03:01:10.498365 29915 leader_election.cc:302] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c27dbf67a1f4d699532bc3d457baffe, cf3e070fee0e43f597a8d5c466f3a095; no voters:
I20260807 03:01:10.498685 30468 raft_consensus.cc:2781] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:10.499187 30468 raft_consensus.cc:686] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 1 LEADER]: Becoming Leader. State: Replica: cf3e070fee0e43f597a8d5c466f3a095, State: Running, Role: LEADER
I20260807 03:01:10.499677 30468 consensus_queue.cc:237] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } }
I20260807 03:01:10.504127 29814 catalog_manager.cc:5495] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf3e070fee0e43f597a8d5c466f3a095 (127.1.121.65). New cstate: current_term: 1 leader_uuid: "cf3e070fee0e43f597a8d5c466f3a095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf3e070fee0e43f597a8d5c466f3a095" member_type: VOTER last_known_addr { host: "127.1.121.65" port: 37197 } health_report { overall_health: HEALTHY } } }
W20260807 03:01:10.510634 30433 tablet.cc:2369] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:01:10.512737 30255 tablet.cc:2369] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:01:10.518429 30069 tablet.cc:2369] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:01:10.522071 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.516778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.1.121.65:37197)
0807 03:01:10.516982 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.516987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.517114 (+ 127us) server_negotiation.cc:407] Connection header received
0807 03:01:10.517303 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.517307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.517396 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.517523 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:01:10.517853 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.518578 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.520156 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.520561 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.520664 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.521148 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:01:10.521631 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:01:10.521755 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.521827 (+ 72us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.521890 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 03:01:10.530216 30366 raft_consensus.cc:1261] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [term 1 FOLLOWER]: Refusing update from remote peer cf3e070fee0e43f597a8d5c466f3a095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:01:10.530215 30190 raft_consensus.cc:1261] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [term 1 FOLLOWER]: Refusing update from remote peer cf3e070fee0e43f597a8d5c466f3a095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:01:10.531270 30474 consensus_queue.cc:1035] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c27dbf67a1f4d699532bc3d457baffe" member_type: VOTER last_known_addr { host: "127.1.121.67" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:01:10.531625 30468 consensus_queue.cc:1035] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2680c59f224624b4af07e1058646c1" member_type: VOTER last_known_addr { host: "127.1.121.66" port: 38829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:01:10.542834 30479 mvcc.cc:205] Tried to move back new op lower bound from 7315749562481324032 to 7315749562370441216. Current Snapshot: MvccSnapshot[applied={T|T < 7315749562481324032}]
I20260807 03:01:10.544130 30483 mvcc.cc:205] Tried to move back new op lower bound from 7315749562481324032 to 7315749562370441216. Current Snapshot: MvccSnapshot[applied={T|T < 7315749562481324032}]
I20260807 03:01:10.558431 29814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:01:10.558706 29814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:01:10.562441 29814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 289934dd78a040d2bb53368d59c0d5cf: Sending DeleteTablet for 3 replicas of tablet 3fe7642172964a28b092e8786b1cbf95
I20260807 03:01:10.563817 30341 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe7642172964a28b092e8786b1cbf95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:01:10 UTC) from {username='slave'} at 127.0.0.1:56198
I20260807 03:01:10.563958 30168 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe7642172964a28b092e8786b1cbf95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:01:10 UTC) from {username='slave'} at 127.0.0.1:36650
I20260807 03:01:10.563992 29978 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe7642172964a28b092e8786b1cbf95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:01:10 UTC) from {username='slave'} at 127.0.0.1:53374
I20260807 03:01:10.565865 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 29892
I20260807 03:01:10.578809 30490 tablet_replica.cc:330] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: stopping tablet replica
I20260807 03:01:10.579494 30491 tablet_replica.cc:330] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: stopping tablet replica
I20260807 03:01:10.579993 30492 tablet_replica.cc:330] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: stopping tablet replica
I20260807 03:01:10.580152 30490 raft_consensus.cc:2229] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:01:10.580200 30491 raft_consensus.cc:2229] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:01:10.580421 30492 raft_consensus.cc:2229] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:01:10.580758 30490 raft_consensus.cc:2258] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:01:10.580961 30491 raft_consensus.cc:2258] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:01:10.581218 30492 raft_consensus.cc:2258] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:01:10.583652 30492 ts_tablet_manager.cc:1905] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:01:10.583652 30490 ts_tablet_manager.cc:1905] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:01:10.583716 30491 ts_tablet_manager.cc:1905] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:01:10.590113 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.567063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.1.121.65:37197)
0807 03:01:10.567267 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.567273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.570325 (+ 3052us) server_negotiation.cc:407] Connection header received
0807 03:01:10.581297 (+ 10972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.581309 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.581434 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.581565 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:10.582152 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.583243 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.585177 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.585848 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.585983 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.587586 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:10.587603 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:10.587623 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:10.588525 (+ 902us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:10.588591 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:10.588625 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:10.588751 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.589426 (+ 675us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.589547 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:01:10.590799 30061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:10.591403 30490 ts_tablet_manager.cc:1918] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:01:10.591590 30490 log.cc:1196] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-2/wal/wals/3fe7642172964a28b092e8786b1cbf95
I20260807 03:01:10.592389 30491 ts_tablet_manager.cc:1918] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:01:10.592417 30490 ts_tablet_manager.cc:1939] T 3fe7642172964a28b092e8786b1cbf95 P 8c27dbf67a1f4d699532bc3d457baffe: Deleting consensus metadata
I20260807 03:01:10.592553 30491 log.cc:1196] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-1/wal/wals/3fe7642172964a28b092e8786b1cbf95
I20260807 03:01:10.593169 30491 ts_tablet_manager.cc:1939] T 3fe7642172964a28b092e8786b1cbf95 P 4e2680c59f224624b4af07e1058646c1: Deleting consensus metadata
I20260807 03:01:10.594470 29789 catalog_manager.cc:4841] TS 8c27dbf67a1f4d699532bc3d457baffe (127.1.121.67:40883): tablet 3fe7642172964a28b092e8786b1cbf95 (table test-table [id=112e150bda144b6b9683397d6281b701]) successfully deleted
I20260807 03:01:10.595227 29793 catalog_manager.cc:4841] TS 4e2680c59f224624b4af07e1058646c1 (127.1.121.66:38829): tablet 3fe7642172964a28b092e8786b1cbf95 (table test-table [id=112e150bda144b6b9683397d6281b701]) successfully deleted
I20260807 03:01:10.701387 30492 ts_tablet_manager.cc:1918] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:01:10.701709 30492 log.cc:1196] T 3fe7642172964a28b092e8786b1cbf95 P cf3e070fee0e43f597a8d5c466f3a095: Deleting WAL directory at /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786071323612974-1509-0/minicluster-data/ts-0/wal/wals/3fe7642172964a28b092e8786b1cbf95
I20260807 03:01:10.702072 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 29892
I20260807 03:01:10.728303 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 30073
W20260807 03:01:10.728298 29789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.121.65:37197 (error 108)
I20260807 03:01:10.730428 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.729894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37197 (local address 127.0.0.1:53414)
0807 03:01:10.730105 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.730245 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 03:01:10.730732 29789 catalog_manager.cc:4584] TS cf3e070fee0e43f597a8d5c466f3a095 (127.1.121.65:37197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fe7642172964a28b092e8786b1cbf95: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37197: connect: Connection refused (error 111)
I20260807 03:01:10.735684 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.729278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.1.121.66:38829)
0807 03:01:10.729461 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.729466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.729592 (+ 126us) server_negotiation.cc:407] Connection header received
0807 03:01:10.729837 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.729843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.729966 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.730088 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:10.730576 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.731433 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.732644 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.733243 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.733338 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.734370 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:10.734378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:10.734392 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:10.734968 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:10.735019 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:10.735038 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:10.735123 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.735245 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.735330 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 03:01:10.736486 30249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:10.784344 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.783773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37197 (local address 127.0.0.1:53426)
0807 03:01:10.784005 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.784141 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 03:01:10.785476 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.785075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37197 (local address 127.0.0.1:53440)
0807 03:01:10.785235 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.785321 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 03:01:10.829027 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 30073
I20260807 03:01:10.846114 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.845637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37197 (local address 127.0.0.1:53442)
0807 03:01:10.845851 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.845938 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 03:01:10.847240 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.846887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37197 (local address 127.0.0.1:53456)
0807 03:01:10.847052 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.847126 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 03:01:10.854228 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 30260
I20260807 03:01:10.860741 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.855033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.1.121.67:40883)
0807 03:01:10.855218 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:10.855222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:10.855240 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:01:10.855396 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:10.855400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:10.855487 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:10.855575 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:10.855954 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.856691 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.857994 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:10.858366 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:10.858462 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:10.859425 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:10.859434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:10.859451 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:10.860030 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:10.860078 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:10.860097 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:10.860188 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:10.860306 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 03:01:10.860391 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:01:10.861429 30424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:10.928099 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.927380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37197 (local address 127.0.0.1:53458)
0807 03:01:10.927702 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.927841 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260807 03:01:10.929519 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:10.928953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37197 (local address 127.0.0.1:53462)
0807 03:01:10.929206 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:10.929318 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260807 03:01:10.958772 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 30260
I20260807 03:01:10.988900 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 29761
I20260807 03:01:10.990137 29852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:11.102895 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:11.102009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.65:37197 (local address 127.0.0.1:53476)
0807 03:01:11.102226 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:11.102349 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.65:37197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 03:01:11.103389 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 29761
2026-08-07T03:01:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3058 ms)
[----------] 6 tests from EncryptionPolicyTest (24297 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:11 dist-test-slave-3h5h krb5kdc[30505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:11 dist-test-slave-3h5h krb5kdc[30505](info): set up 2 sockets
Aug 07 03:01:11 dist-test-slave-3h5h krb5kdc[30505](info): commencing operation
krb5kdc: starting...
W20260807 03:01:12.464172 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.005s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:12 dist-test-slave-3h5h krb5kdc[30505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:12.817725 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:13.196949 30526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:13.197299 30526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:13.197398 30526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:13.205549 30526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:13.205680 30526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:13.205729 30526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:13.205792 30526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:13.205838 30526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:13.217163 30526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:13.218580 30526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:13.219920 30526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:13.232573 30536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:13.232537 30535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:13.233366 30538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:13.233996 30526 server_base.cc:1034] running on GCE node
I20260807 03:01:13.234892 30526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:13.236665 30526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:13.238016 30526 hybrid_clock.cc:648] HybridClock initialized: now 1786071673237952 us; error 78 us; skew 500 ppm
I20260807 03:01:13.238435 30526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:13 dist-test-slave-3h5h krb5kdc[30505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:13.248436 30526 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:13.250296 30526 webserver.cc:460] Webserver started at http://127.0.0.1:38763/ using document root <none> and password file <none>
I20260807 03:01:13.251106 30526 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:13.251264 30526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:13.251595 30526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:13.254832 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "be6453f1f888445d8b89979dae0d5f60"
format_stamp: "Formatted at 2026-08-07 03:01:13 on dist-test-slave-3h5h"
server_key: "c2c2577623448f2d01c9eebcaec3907d"
server_key_iv: "133fd1420b68221f7d776f785e7aebd3"
server_key_version: "encryptionkey@0"
I20260807 03:01:13.255635 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "be6453f1f888445d8b89979dae0d5f60"
format_stamp: "Formatted at 2026-08-07 03:01:13 on dist-test-slave-3h5h"
server_key: "c2c2577623448f2d01c9eebcaec3907d"
server_key_iv: "133fd1420b68221f7d776f785e7aebd3"
server_key_version: "encryptionkey@0"
I20260807 03:01:13.261310 30526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 03:01:13.264892 30545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:13.266254 30526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:01:13.266459 30526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "be6453f1f888445d8b89979dae0d5f60"
format_stamp: "Formatted at 2026-08-07 03:01:13 on dist-test-slave-3h5h"
server_key: "c2c2577623448f2d01c9eebcaec3907d"
server_key_iv: "133fd1420b68221f7d776f785e7aebd3"
server_key_version: "encryptionkey@0"
I20260807 03:01:13.266682 30526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:13.283077 30526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:13.287535 30526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:13.287861 30526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:13.309676 30526 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44621
I20260807 03:01:13.309700 30618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44621 every 8 connection(s)
I20260807 03:01:13.311297 30526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:13.313550 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 30526
I20260807 03:01:13.313802 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:13.314672 1509 external_mini_cluster.cc:1428] Setting key e8e87d5c096ea5072be3c49684e9ba57
I20260807 03:01:13.316344 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 03:01:13 dist-test-slave-3h5h krb5kdc[30505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 03:01:13.331537 30619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60: Bootstrap starting.
I20260807 03:01:13.335110 30619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:13.336551 30619 log.cc:824] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:13.339603 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:13.316795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37679 (local address 127.0.0.1:44621)
0807 03:01:13.317688 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:13.317700 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:13.317744 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:01:13.319359 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:13.319381 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:13.319790 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:13.320135 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:13.322041 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:13.323441 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:13.325257 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:13.325963 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:13.326099 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:13.331483 (+ 5384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:13.331492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:13.331508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:13.331552 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:13.334838 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:13.335919 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:13.335927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:13.335934 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:13.336099 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:13.336648 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:13.336651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:13.336653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:13.337444 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:13.337848 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:13.338218 (+ 370us) server_negotiation.cc:299] Negotiation successful
0807 03:01:13.338600 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":383,"threads_started":1}
I20260807 03:01:13.339917 30619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60: No bootstrap required, opened a new log
I20260807 03:01:13.343699 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:13.350363 30619 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6453f1f888445d8b89979dae0d5f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } }
I20260807 03:01:13.350894 30619 raft_consensus.cc:374] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:13.351012 30619 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6453f1f888445d8b89979dae0d5f60, State: Initialized, Role: FOLLOWER
I20260807 03:01:13.351637 30619 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6453f1f888445d8b89979dae0d5f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } }
I20260807 03:01:13.351889 30619 raft_consensus.cc:388] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:13.352052 30619 raft_consensus.cc:482] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:13.352246 30619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:13.355796 30619 raft_consensus.cc:504] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6453f1f888445d8b89979dae0d5f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } }
I20260807 03:01:13.356493 30619 leader_election.cc:302] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be6453f1f888445d8b89979dae0d5f60; no voters:
I20260807 03:01:13.359007 30619 leader_election.cc:288] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:13.359115 30631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:13.360625 30631 raft_consensus.cc:686] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [term 1 LEADER]: Becoming Leader. State: Replica: be6453f1f888445d8b89979dae0d5f60, State: Running, Role: LEADER
I20260807 03:01:13.361327 30631 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6453f1f888445d8b89979dae0d5f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } }
I20260807 03:01:13.362408 30619 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:13.365464 30633 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6453f1f888445d8b89979dae0d5f60. Latest consensus state: current_term: 1 leader_uuid: "be6453f1f888445d8b89979dae0d5f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6453f1f888445d8b89979dae0d5f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } } }
I20260807 03:01:13.365893 30633 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:13.365625 30634 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be6453f1f888445d8b89979dae0d5f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6453f1f888445d8b89979dae0d5f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } } }
I20260807 03:01:13.369160 30634 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:13.371752 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:13.349865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44501 (local address 127.0.0.1:44621)
0807 03:01:13.350070 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:13.350078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:13.350548 (+ 470us) server_negotiation.cc:407] Connection header received
0807 03:01:13.350685 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:13.350690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:13.350778 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:13.350903 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:13.352864 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:13.354127 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:13.355890 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:13.356616 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:13.356747 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:13.360343 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:13.360349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:13.360355 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:13.360409 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:13.364486 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:13.365770 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:13.365778 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:13.365782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:13.365996 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:13.366648 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:13.366657 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:13.366663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:13.369490 (+ 2827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:13.370525 (+ 1035us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:13.370916 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 03:01:13.371197 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:01:13.375175 30640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:13.381583 30640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:13.391451 30640 catalog_manager.cc:1347] Generated new cluster ID: a0e02c06db614b15bd2aa9c5e8bd2d75
I20260807 03:01:13.391615 30640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:13.403427 30640 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:13.405323 30640 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:13.419850 30640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be6453f1f888445d8b89979dae0d5f60: Generated new TSK 0
I20260807 03:01:13.420917 30640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:13.436702 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 30526
I20260807 03:01:13.438336 30612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:13.531404 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 30526
W20260807 03:01:13.556471 30503 connection.cc:535] client connection to 127.0.0.1:44621 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44621): Connection reset by peer (error 104)
2026-08-07T03:01:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:13 dist-test-slave-3h5h krb5kdc[30662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:13 dist-test-slave-3h5h krb5kdc[30662](info): set up 2 sockets
Aug 07 03:01:13 dist-test-slave-3h5h krb5kdc[30662](info): commencing operation
krb5kdc: starting...
W20260807 03:01:14.940568 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.365s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:15 dist-test-slave-3h5h krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:15.305637 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:15.661770 30683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:15.662166 30683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:15.662357 30683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:15.669853 30683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:15.670009 30683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:15.670121 30683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:15.670233 30683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:15.670321 30683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:15.680456 30683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:15.681891 30683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:15.683283 30683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:15.695420 30694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:15.695483 30695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:15.695901 30697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:15.696254 30683 server_base.cc:1034] running on GCE node
I20260807 03:01:15.697222 30683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:15.698699 30683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:15.699937 30683 hybrid_clock.cc:648] HybridClock initialized: now 1786071675699881 us; error 53 us; skew 500 ppm
I20260807 03:01:15.700353 30683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:15 dist-test-slave-3h5h krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:15.703703 30683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:15.705420 30683 webserver.cc:460] Webserver started at http://127.0.0.1:41421/ using document root <none> and password file <none>
I20260807 03:01:15.706171 30683 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:15.706324 30683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:15.706651 30683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:15.709795 30683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "39437962b5424562938b0d45b8ee3b0f"
format_stamp: "Formatted at 2026-08-07 03:01:15 on dist-test-slave-3h5h"
server_key: "745e4ab93f21d3695f36ee58322fcb3e"
server_key_iv: "e9ab2a62cd620168aa20db10cae18814"
server_key_version: "encryptionkey@0"
I20260807 03:01:15.710599 30683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "39437962b5424562938b0d45b8ee3b0f"
format_stamp: "Formatted at 2026-08-07 03:01:15 on dist-test-slave-3h5h"
server_key: "745e4ab93f21d3695f36ee58322fcb3e"
server_key_iv: "e9ab2a62cd620168aa20db10cae18814"
server_key_version: "encryptionkey@0"
I20260807 03:01:15.716135 30683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 03:01:15.719905 30706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:15.721401 30683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:01:15.721602 30683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "39437962b5424562938b0d45b8ee3b0f"
format_stamp: "Formatted at 2026-08-07 03:01:15 on dist-test-slave-3h5h"
server_key: "745e4ab93f21d3695f36ee58322fcb3e"
server_key_iv: "e9ab2a62cd620168aa20db10cae18814"
server_key_version: "encryptionkey@0"
I20260807 03:01:15.721817 30683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:15.745426 30683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:15.749862 30683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:15.750190 30683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:15.770195 30683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46551
I20260807 03:01:15.770303 30783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46551 every 8 connection(s)
I20260807 03:01:15.771679 30683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:15.775753 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:15.780825 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 30683
I20260807 03:01:15.781082 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:15.781659 1509 external_mini_cluster.cc:1428] Setting key 5e746093150bf943751cc4721805e114
I20260807 03:01:15.788028 30784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f: Bootstrap starting.
I20260807 03:01:15.791898 30784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:15.793411 30784 log.cc:824] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:15.796392 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:15.783636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51241 (local address 127.0.0.1:46551)
0807 03:01:15.784495 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:15.784507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:15.784562 (+ 55us) server_negotiation.cc:407] Connection header received
0807 03:01:15.786474 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:15.786500 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:15.786924 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:15.787261 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:15.787916 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:15.789411 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:15.791362 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:15.792146 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:15.792285 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:15.793725 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:15.793733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:15.793748 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:15.794558 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:15.794661 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:15.794708 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:15.794890 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:15.795184 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 03:01:15.795393 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":399,"threads_started":1}
I20260807 03:01:15.796746 30784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f: No bootstrap required, opened a new log
I20260807 03:01:15.800513 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:15.807011 30784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39437962b5424562938b0d45b8ee3b0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46551 } }
I20260807 03:01:15.807503 30784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:15.807642 30784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39437962b5424562938b0d45b8ee3b0f, State: Initialized, Role: FOLLOWER
I20260807 03:01:15.808386 30784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39437962b5424562938b0d45b8ee3b0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46551 } }
I20260807 03:01:15.808706 30784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:15.808912 30784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:15.809135 30784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:15.812441 30784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39437962b5424562938b0d45b8ee3b0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46551 } }
I20260807 03:01:15.813160 30784 leader_election.cc:302] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39437962b5424562938b0d45b8ee3b0f; no voters:
I20260807 03:01:15.813972 30784 leader_election.cc:288] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:15.814193 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:15.806031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37639 (local address 127.0.0.1:46551)
0807 03:01:15.806267 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:15.806274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:15.806713 (+ 439us) server_negotiation.cc:407] Connection header received
0807 03:01:15.806854 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:15.806859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:15.806938 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:15.807074 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:15.807660 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:15.808768 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:15.810631 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:15.811352 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:15.811484 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:15.813007 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:15.813013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:15.813018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:15.813040 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:15.813137 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:15.813180 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:15.813282 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:15.813605 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 03:01:15.813702 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 03:01:15.814193 30801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:15.815974 30801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [term 1 LEADER]: Becoming Leader. State: Replica: 39437962b5424562938b0d45b8ee3b0f, State: Running, Role: LEADER
I20260807 03:01:15.816550 30784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:15.816514 30801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39437962b5424562938b0d45b8ee3b0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46551 } }
I20260807 03:01:15.820096 30803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39437962b5424562938b0d45b8ee3b0f. Latest consensus state: current_term: 1 leader_uuid: "39437962b5424562938b0d45b8ee3b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39437962b5424562938b0d45b8ee3b0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46551 } } }
I20260807 03:01:15.819953 30802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39437962b5424562938b0d45b8ee3b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39437962b5424562938b0d45b8ee3b0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46551 } } }
I20260807 03:01:15.820511 30803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:15.820566 30802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:15.823005 30810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:15.830379 30810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:15.841388 30810 catalog_manager.cc:1347] Generated new cluster ID: 9a0589c4ac7a4f0cafb37e3410bed704
I20260807 03:01:15.841519 30810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:15.873740 30810 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:15.875176 30810 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:15.883436 30810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39437962b5424562938b0d45b8ee3b0f: Generated new TSK 0
I20260807 03:01:15.884253 30810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:15.900449 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 30683
W20260807 03:01:15.901697 30777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51241
W20260807 03:01:15.902529 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 30683: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:15.902724 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 30683
W20260807 03:01:15.929610 30660 connection.cc:535] client connection to 127.0.0.1:46551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46551): Connection reset by peer (error 104)
2026-08-07T03:01:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:16 dist-test-slave-3h5h krb5kdc[30824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:16 dist-test-slave-3h5h krb5kdc[30824](info): set up 2 sockets
Aug 07 03:01:16 dist-test-slave-3h5h krb5kdc[30824](info): commencing operation
krb5kdc: starting...
W20260807 03:01:17.233238 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.004s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:17 dist-test-slave-3h5h krb5kdc[30824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:17.582504 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:17.939529 30839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:17.939908 30839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:17.940083 30839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:17.948387 30839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:17.948601 30839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:17.948730 30839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:17.948931 30839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:17.949031 30839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:17.960518 30839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:17.962051 30839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:17.963505 30839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:17.977206 30849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:17.977206 30852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:17.977239 30850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:17.978293 30839 server_base.cc:1034] running on GCE node
I20260807 03:01:17.979136 30839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:17.980661 30839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:17.981894 30839 hybrid_clock.cc:648] HybridClock initialized: now 1786071677981836 us; error 53 us; skew 500 ppm
I20260807 03:01:17.982275 30839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:17 dist-test-slave-3h5h krb5kdc[30824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:17.985460 30839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:17.986918 30839 webserver.cc:460] Webserver started at http://127.0.0.1:44343/ using document root <none> and password file <none>
I20260807 03:01:17.987596 30839 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:17.987699 30839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:17.987955 30839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:17.990928 30839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "0367d2ba06554bb5a689c32598f78bfd"
format_stamp: "Formatted at 2026-08-07 03:01:17 on dist-test-slave-3h5h"
server_key: "37e0173b9c21799e0df0cf2fbe15fac1"
server_key_iv: "c6247e9e265bd2ed9c317f9ce68460e5"
server_key_version: "encryptionkey@0"
I20260807 03:01:17.991676 30839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "0367d2ba06554bb5a689c32598f78bfd"
format_stamp: "Formatted at 2026-08-07 03:01:17 on dist-test-slave-3h5h"
server_key: "37e0173b9c21799e0df0cf2fbe15fac1"
server_key_iv: "c6247e9e265bd2ed9c317f9ce68460e5"
server_key_version: "encryptionkey@0"
I20260807 03:01:17.996809 30839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260807 03:01:18.000514 30867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:18.001979 30839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:01:18.002173 30839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "0367d2ba06554bb5a689c32598f78bfd"
format_stamp: "Formatted at 2026-08-07 03:01:17 on dist-test-slave-3h5h"
server_key: "37e0173b9c21799e0df0cf2fbe15fac1"
server_key_iv: "c6247e9e265bd2ed9c317f9ce68460e5"
server_key_version: "encryptionkey@0"
I20260807 03:01:18.002388 30839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:18.008939 30839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:18.013202 30839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:18.013509 30839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:18.033238 30839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39113
I20260807 03:01:18.033316 30936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39113 every 8 connection(s)
I20260807 03:01:18.034806 30839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:18.037218 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 30839
I20260807 03:01:18.037391 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:18.037858 1509 external_mini_cluster.cc:1428] Setting key 1dca3d11b60b53b427dae505943fd0eb
I20260807 03:01:18.039208 30937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:18.045907 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:18.039724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50633 (local address 127.0.0.1:39113)
0807 03:01:18.040519 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:18.040532 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:18.040573 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:01:18.042202 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:18.042225 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:18.042565 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:18.042919 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:18.044129 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:18.044135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:18.044149 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:18.044173 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:18.044258 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:18.044283 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:18.044469 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:18.044636 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 03:01:18.044928 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":361,"threads_started":1}
I20260807 03:01:18.049746 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:18.053462 30937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd: Bootstrap starting.
I20260807 03:01:18.058086 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:18.055562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53705 (local address 127.0.0.1:39113)
0807 03:01:18.055747 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:18.055752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:18.056036 (+ 284us) server_negotiation.cc:407] Connection header received
0807 03:01:18.056251 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:18.056255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:18.056304 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:18.056377 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:18.057404 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:18.057408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:18.057411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:18.057425 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:18.057463 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:18.057478 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:18.057566 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:18.057713 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 03:01:18.057762 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:01:18.058122 30937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:18.059383 30937 log.cc:824] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:18.062463 30937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd: No bootstrap required, opened a new log
I20260807 03:01:18.070355 30937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0367d2ba06554bb5a689c32598f78bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260807 03:01:18.070787 30937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:18.070904 30937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0367d2ba06554bb5a689c32598f78bfd, State: Initialized, Role: FOLLOWER
I20260807 03:01:18.071817 30937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0367d2ba06554bb5a689c32598f78bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260807 03:01:18.072201 30937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:18.072381 30937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:18.072589 30937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:18.075374 30937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0367d2ba06554bb5a689c32598f78bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260807 03:01:18.075951 30937 leader_election.cc:302] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0367d2ba06554bb5a689c32598f78bfd; no voters:
I20260807 03:01:18.076633 30937 leader_election.cc:288] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:18.076747 30950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:18.077641 30950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [term 1 LEADER]: Becoming Leader. State: Replica: 0367d2ba06554bb5a689c32598f78bfd, State: Running, Role: LEADER
I20260807 03:01:18.078272 30950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0367d2ba06554bb5a689c32598f78bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260807 03:01:18.079157 30937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:18.081562 30951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0367d2ba06554bb5a689c32598f78bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0367d2ba06554bb5a689c32598f78bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260807 03:01:18.082039 30951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:18.081655 30953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0367d2ba06554bb5a689c32598f78bfd. Latest consensus state: current_term: 1 leader_uuid: "0367d2ba06554bb5a689c32598f78bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0367d2ba06554bb5a689c32598f78bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260807 03:01:18.082665 30953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:18.083528 30957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:18.089293 30957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:18.099304 30957 catalog_manager.cc:1347] Generated new cluster ID: 4ef866d8c4aa42e5ab931ed0dac12665
I20260807 03:01:18.099449 30957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:18.122881 30957 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:18.123379 30957 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:18.131639 30957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0367d2ba06554bb5a689c32598f78bfd: Generated new TSK 0
I20260807 03:01:18.132521 30957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:18.141618 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 30839
W20260807 03:01:18.142795 30927 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50633
W20260807 03:01:18.143560 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 30839: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:18.143739 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 30839
2026-08-07T03:01:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:18 dist-test-slave-3h5h krb5kdc[30979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:18 dist-test-slave-3h5h krb5kdc[30979](info): set up 2 sockets
Aug 07 03:01:18 dist-test-slave-3h5h krb5kdc[30979](info): commencing operation
krb5kdc: starting...
W20260807 03:01:19.603364 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.414s user 0.002s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:19 dist-test-slave-3h5h krb5kdc[30979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:19.957435 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:20.333371 30998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:20.333735 30998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:20.333917 30998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:20.341249 30998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:20.341418 30998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:20.341538 30998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:20.341661 30998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:20.341754 30998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:20.351926 30998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:20.353399 30998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:20.354799 30998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:20.367451 31008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:20.367616 31009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:20.367630 31015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:20.368979 30998 server_base.cc:1034] running on GCE node
I20260807 03:01:20.369912 30998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:20.371485 30998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:20.372751 30998 hybrid_clock.cc:648] HybridClock initialized: now 1786071680372699 us; error 66 us; skew 500 ppm
I20260807 03:01:20.373167 30998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:20 dist-test-slave-3h5h krb5kdc[30979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:20.376528 30998 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:20.378189 30998 webserver.cc:460] Webserver started at http://127.0.0.1:44051/ using document root <none> and password file <none>
I20260807 03:01:20.378896 30998 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:20.379006 30998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:20.379269 30998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:20.382401 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "f26c20f834274c7cbb7026855dfba100"
format_stamp: "Formatted at 2026-08-07 03:01:20 on dist-test-slave-3h5h"
server_key: "a86c6ded078ac1d5619a652f190e2a3f"
server_key_iv: "1c3dac027f508b1f3f37dd96d2168d0a"
server_key_version: "encryptionkey@0"
I20260807 03:01:20.383162 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "f26c20f834274c7cbb7026855dfba100"
format_stamp: "Formatted at 2026-08-07 03:01:20 on dist-test-slave-3h5h"
server_key: "a86c6ded078ac1d5619a652f190e2a3f"
server_key_iv: "1c3dac027f508b1f3f37dd96d2168d0a"
server_key_version: "encryptionkey@0"
I20260807 03:01:20.388659 30998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:01:20.392315 31025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:20.393970 30998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:01:20.394208 30998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "f26c20f834274c7cbb7026855dfba100"
format_stamp: "Formatted at 2026-08-07 03:01:20 on dist-test-slave-3h5h"
server_key: "a86c6ded078ac1d5619a652f190e2a3f"
server_key_iv: "1c3dac027f508b1f3f37dd96d2168d0a"
server_key_version: "encryptionkey@0"
I20260807 03:01:20.394429 30998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:20.424317 30998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:20.429461 30998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:20.429843 30998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:20.451121 30998 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43983
I20260807 03:01:20.451146 31093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43983 every 8 connection(s)
I20260807 03:01:20.452694 30998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:20.453881 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 30998
I20260807 03:01:20.454171 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:20.454638 1509 external_mini_cluster.cc:1428] Setting key 824647c72da0ebff4bb04f0533240015
I20260807 03:01:20.457445 31094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 03:01:20 dist-test-slave-3h5h krb5kdc[30979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 03:01:20.471184 31094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100: Bootstrap starting.
I20260807 03:01:20.476227 31094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:20.477736 31094 log.cc:824] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:20.481669 31094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100: No bootstrap required, opened a new log
I20260807 03:01:20.481877 31097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:20.456678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34523 (local address 127.0.0.1:43983)
0807 03:01:20.457616 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:20.457631 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:20.457684 (+ 53us) server_negotiation.cc:407] Connection header received
0807 03:01:20.460022 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:20.460052 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:20.460576 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:20.461069 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:20.462645 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:20.464685 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:20.466260 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:20.467344 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:20.467544 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:20.472126 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:20.472137 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:20.472156 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:20.472206 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:20.476908 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:20.477658 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:20.477670 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:20.477682 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:20.477958 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:20.478451 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:20.478460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:20.478466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:20.479128 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:20.479455 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:20.479912 (+ 457us) server_negotiation.cc:299] Negotiation successful
0807 03:01:20.480387 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":413,"threads_started":1}
I20260807 03:01:20.485929 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:20.491158 31094 raft_consensus.cc:348] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26c20f834274c7cbb7026855dfba100" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260807 03:01:20.491580 31094 raft_consensus.cc:374] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:20.491717 31094 raft_consensus.cc:729] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26c20f834274c7cbb7026855dfba100, State: Initialized, Role: FOLLOWER
I20260807 03:01:20.492391 31094 consensus_queue.cc:260] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26c20f834274c7cbb7026855dfba100" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260807 03:01:20.492674 31094 raft_consensus.cc:388] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:20.492839 31094 raft_consensus.cc:482] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:20.493111 31094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:20.495673 31094 raft_consensus.cc:504] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26c20f834274c7cbb7026855dfba100" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260807 03:01:20.496246 31094 leader_election.cc:302] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f26c20f834274c7cbb7026855dfba100; no voters:
I20260807 03:01:20.497143 31094 leader_election.cc:288] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:20.497298 31106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:20.498771 31106 raft_consensus.cc:686] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [term 1 LEADER]: Becoming Leader. State: Replica: f26c20f834274c7cbb7026855dfba100, State: Running, Role: LEADER
I20260807 03:01:20.499619 31106 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26c20f834274c7cbb7026855dfba100" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260807 03:01:20.499933 31094 sys_catalog.cc:564] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:20.502851 31107 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f26c20f834274c7cbb7026855dfba100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26c20f834274c7cbb7026855dfba100" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } } }
I20260807 03:01:20.503386 31107 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:20.508252 31108 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f26c20f834274c7cbb7026855dfba100. Latest consensus state: current_term: 1 leader_uuid: "f26c20f834274c7cbb7026855dfba100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26c20f834274c7cbb7026855dfba100" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } } }
I20260807 03:01:20.509326 31097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:20.490347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38621 (local address 127.0.0.1:43983)
0807 03:01:20.490527 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:20.490531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:20.491069 (+ 538us) server_negotiation.cc:407] Connection header received
0807 03:01:20.491263 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:20.491268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:20.491336 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:20.491442 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:20.493199 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:20.494001 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:20.495224 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:20.495708 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:20.495850 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:20.499348 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:20.499377 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:20.499382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:20.499455 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:20.504194 (+ 4739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:20.505254 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:20.505263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:20.505269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:20.505467 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:20.506083 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:20.506090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:20.506096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:20.506601 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:20.508246 (+ 1645us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:20.508455 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 03:01:20.508755 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 03:01:20.511294 31108 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:20.515404 31111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:20.521330 31111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:20.532721 31111 catalog_manager.cc:1347] Generated new cluster ID: 9fc3c22fb9344112a84ad93272231a88
I20260807 03:01:20.532904 31111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:20.583122 31111 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:20.584656 31111 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:20.591208 31111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f26c20f834274c7cbb7026855dfba100: Generated new TSK 0
I20260807 03:01:20.592087 31111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:20.606964 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 30998
I20260807 03:01:20.608134 31087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:20.705678 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 30998
2026-08-07T03:01:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2562 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:20 dist-test-slave-3h5h krb5kdc[31143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:20 dist-test-slave-3h5h krb5kdc[31143](info): set up 2 sockets
Aug 07 03:01:20 dist-test-slave-3h5h krb5kdc[31143](info): commencing operation
krb5kdc: starting...
W20260807 03:01:22.148937 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.397s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:22 dist-test-slave-3h5h krb5kdc[31143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:22.502511 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:22.855989 31159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:22.856361 31159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:22.856526 31159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:22.863888 31159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:22.864051 31159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:22.864163 31159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:22.864269 31159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:22.864347 31159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:22.874514 31159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:22.875880 31159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:22.877266 31159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:22.890004 31164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:22.889972 31170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:22.890057 31168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:22.890692 31159 server_base.cc:1034] running on GCE node
I20260807 03:01:22.891503 31159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:22.893030 31159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:22.894268 31159 hybrid_clock.cc:648] HybridClock initialized: now 1786071682894214 us; error 43 us; skew 500 ppm
I20260807 03:01:22.894631 31159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:22 dist-test-slave-3h5h krb5kdc[31143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:22.898072 31159 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:22.899533 31159 webserver.cc:460] Webserver started at http://127.0.0.1:45311/ using document root <none> and password file <none>
I20260807 03:01:22.900246 31159 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:22.900350 31159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:22.900609 31159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:22.903743 31159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "76ffbd9c09e248988e6a685769c3c05a"
format_stamp: "Formatted at 2026-08-07 03:01:22 on dist-test-slave-3h5h"
server_key: "e2a97acb2b8ae7be0cd8b9a272ff3074"
server_key_iv: "bc6009e35277b619e8fd588d044057c6"
server_key_version: "encryptionkey@0"
I20260807 03:01:22.904482 31159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "76ffbd9c09e248988e6a685769c3c05a"
format_stamp: "Formatted at 2026-08-07 03:01:22 on dist-test-slave-3h5h"
server_key: "e2a97acb2b8ae7be0cd8b9a272ff3074"
server_key_iv: "bc6009e35277b619e8fd588d044057c6"
server_key_version: "encryptionkey@0"
I20260807 03:01:22.909976 31159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260807 03:01:22.913534 31177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:22.914983 31159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:01:22.915134 31159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "76ffbd9c09e248988e6a685769c3c05a"
format_stamp: "Formatted at 2026-08-07 03:01:22 on dist-test-slave-3h5h"
server_key: "e2a97acb2b8ae7be0cd8b9a272ff3074"
server_key_iv: "bc6009e35277b619e8fd588d044057c6"
server_key_version: "encryptionkey@0"
I20260807 03:01:22.915306 31159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:22.943662 31159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:22.948498 31159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:22.948788 31159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:22.968587 31159 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35623
I20260807 03:01:22.968612 31251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35623 every 8 connection(s)
I20260807 03:01:22.970126 31159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:22.974289 31253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:22.977809 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 31159
I20260807 03:01:22.978081 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:22.978583 1509 external_mini_cluster.cc:1428] Setting key c88350e101a0cd9426f2938858d51a5e
I20260807 03:01:22.987171 31253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a: Bootstrap starting.
I20260807 03:01:22.991003 31253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:22.992241 31253 log.cc:824] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:22.992854 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:22.980483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42705 (local address 127.0.0.1:35623)
0807 03:01:22.981331 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:22.981342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:22.981387 (+ 45us) server_negotiation.cc:407] Connection header received
0807 03:01:22.983054 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:22.983079 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:22.983491 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:22.983836 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:22.984486 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:22.985982 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:22.987917 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:22.988656 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:22.988801 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:22.990170 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:22.990179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:22.990194 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:22.990770 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:22.990847 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:22.990875 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:22.990964 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:22.991391 (+ 427us) server_negotiation.cc:299] Negotiation successful
0807 03:01:22.991694 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":419,"threads_started":1}
I20260807 03:01:22.995543 31253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a: No bootstrap required, opened a new log
I20260807 03:01:22.996799 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:23.003988 31253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ffbd9c09e248988e6a685769c3c05a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } }
I20260807 03:01:23.004436 31253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:23.004552 31253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ffbd9c09e248988e6a685769c3c05a, State: Initialized, Role: FOLLOWER
I20260807 03:01:23.005251 31253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ffbd9c09e248988e6a685769c3c05a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } }
I20260807 03:01:23.005530 31253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:23.005698 31253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:23.005903 31253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:23.008214 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:23.001564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48339 (local address 127.0.0.1:35623)
0807 03:01:23.001790 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:23.001798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:23.002207 (+ 409us) server_negotiation.cc:407] Connection header received
0807 03:01:23.002436 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:23.002440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:23.002507 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:23.002636 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:23.003178 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:23.003877 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:23.005444 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:23.006111 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:23.006257 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:23.007441 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:23.007452 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:23.007455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:23.007473 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:23.007512 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:23.007528 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:23.007630 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:23.007797 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 03:01:23.007891 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:01:23.008338 31253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ffbd9c09e248988e6a685769c3c05a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } }
I20260807 03:01:23.009013 31253 leader_election.cc:302] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76ffbd9c09e248988e6a685769c3c05a; no voters:
I20260807 03:01:23.009744 31253 leader_election.cc:288] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:23.010151 31266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:23.011055 31266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [term 1 LEADER]: Becoming Leader. State: Replica: 76ffbd9c09e248988e6a685769c3c05a, State: Running, Role: LEADER
I20260807 03:01:23.011632 31266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ffbd9c09e248988e6a685769c3c05a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } }
I20260807 03:01:23.012220 31253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:23.015290 31268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76ffbd9c09e248988e6a685769c3c05a. Latest consensus state: current_term: 1 leader_uuid: "76ffbd9c09e248988e6a685769c3c05a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ffbd9c09e248988e6a685769c3c05a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } } }
I20260807 03:01:23.015347 31267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76ffbd9c09e248988e6a685769c3c05a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ffbd9c09e248988e6a685769c3c05a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } } }
I20260807 03:01:23.015861 31267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:23.015861 31268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:23.017002 31274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:23.022505 31274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:23.032560 31274 catalog_manager.cc:1347] Generated new cluster ID: 7b1d5fc4269042fca13f94878a5de78a
I20260807 03:01:23.032716 31274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:23.060745 31274 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:23.062768 31274 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:23.072258 31274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76ffbd9c09e248988e6a685769c3c05a: Generated new TSK 0
I20260807 03:01:23.073379 31274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:23.088075 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31159
W20260807 03:01:23.089319 31245 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42705
W20260807 03:01:23.090078 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31159: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:23.090225 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 31159
2026-08-07T03:01:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:23 dist-test-slave-3h5h krb5kdc[31299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:23 dist-test-slave-3h5h krb5kdc[31299](info): set up 2 sockets
Aug 07 03:01:23 dist-test-slave-3h5h krb5kdc[31299](info): commencing operation
krb5kdc: starting...
W20260807 03:01:24.417675 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:24 dist-test-slave-3h5h krb5kdc[31299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:24.764289 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:25.131315 31321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:25.131672 31321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:25.131843 31321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:25.139274 31321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:25.139446 31321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:25.139556 31321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:25.139668 31321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:25.139751 31321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:25.149940 31321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:25.151353 31321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:25.152748 31321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:25.166116 31327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:25.166244 31333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:25.166116 31326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:25.167088 31321 server_base.cc:1034] running on GCE node
I20260807 03:01:25.167963 31321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:25.169450 31321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:25.170668 31321 hybrid_clock.cc:648] HybridClock initialized: now 1786071685170601 us; error 48 us; skew 500 ppm
I20260807 03:01:25.170990 31321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:25 dist-test-slave-3h5h krb5kdc[31299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:25.180070 31321 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:25.181883 31321 webserver.cc:460] Webserver started at http://127.0.0.1:43935/ using document root <none> and password file <none>
I20260807 03:01:25.182643 31321 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:25.182787 31321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:25.183122 31321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:25.186293 31321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "877103b851ca4626a1e02bab79b56a0b"
format_stamp: "Formatted at 2026-08-07 03:01:25 on dist-test-slave-3h5h"
server_key: "8df276fc5b3e36e59372339fc76fa3aa"
server_key_iv: "f92f652249d607a4537439c3807c5a10"
server_key_version: "encryptionkey@0"
I20260807 03:01:25.187067 31321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "877103b851ca4626a1e02bab79b56a0b"
format_stamp: "Formatted at 2026-08-07 03:01:25 on dist-test-slave-3h5h"
server_key: "8df276fc5b3e36e59372339fc76fa3aa"
server_key_iv: "f92f652249d607a4537439c3807c5a10"
server_key_version: "encryptionkey@0"
I20260807 03:01:25.192499 31321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 03:01:25.196252 31342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:25.197656 31321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:01:25.197846 31321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "877103b851ca4626a1e02bab79b56a0b"
format_stamp: "Formatted at 2026-08-07 03:01:25 on dist-test-slave-3h5h"
server_key: "8df276fc5b3e36e59372339fc76fa3aa"
server_key_iv: "f92f652249d607a4537439c3807c5a10"
server_key_version: "encryptionkey@0"
I20260807 03:01:25.198055 31321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:25.203927 31321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:25.207868 31321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:25.208180 31321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:25.231709 31321 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43873
I20260807 03:01:25.231745 31412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43873 every 8 connection(s)
I20260807 03:01:25.233875 31321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:25.237883 31414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:25.240440 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 31321
I20260807 03:01:25.240623 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:25.241273 1509 external_mini_cluster.cc:1428] Setting key a7d85cd671141ccfb95819b5ed458980
I20260807 03:01:25.249248 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:25.243227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43539 (local address 127.0.0.1:43873)
0807 03:01:25.244024 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:25.244035 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:25.244082 (+ 47us) server_negotiation.cc:407] Connection header received
0807 03:01:25.245731 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:25.245754 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:25.246077 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:25.246458 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:25.247574 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:25.247581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:25.247593 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:25.247612 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:25.247675 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:25.247696 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:25.247937 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:25.248082 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 03:01:25.248282 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":353,"threads_started":1}
I20260807 03:01:25.252759 31414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b: Bootstrap starting.
I20260807 03:01:25.252955 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:25.256626 31414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:25.258172 31414 log.cc:824] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:25.260051 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:25.257461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34693 (local address 127.0.0.1:43873)
0807 03:01:25.257606 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:25.257610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:25.258120 (+ 510us) server_negotiation.cc:407] Connection header received
0807 03:01:25.258328 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:25.258333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:25.258392 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:25.258490 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:25.259392 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:25.259398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:25.259403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:25.259420 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:25.259464 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:25.259479 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:25.259595 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:25.259724 (+ 129us) server_negotiation.cc:299] Negotiation successful
0807 03:01:25.259788 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 03:01:25.261548 31414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b: No bootstrap required, opened a new log
I20260807 03:01:25.269754 31414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877103b851ca4626a1e02bab79b56a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43873 } }
I20260807 03:01:25.270319 31414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:25.270462 31414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 877103b851ca4626a1e02bab79b56a0b, State: Initialized, Role: FOLLOWER
I20260807 03:01:25.271099 31414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877103b851ca4626a1e02bab79b56a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43873 } }
I20260807 03:01:25.271370 31414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:25.271559 31414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:25.271764 31414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:25.274406 31414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877103b851ca4626a1e02bab79b56a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43873 } }
I20260807 03:01:25.274952 31414 leader_election.cc:302] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 877103b851ca4626a1e02bab79b56a0b; no voters:
I20260807 03:01:25.275592 31414 leader_election.cc:288] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:25.275735 31426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:25.277045 31426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [term 1 LEADER]: Becoming Leader. State: Replica: 877103b851ca4626a1e02bab79b56a0b, State: Running, Role: LEADER
I20260807 03:01:25.277621 31426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877103b851ca4626a1e02bab79b56a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43873 } }
I20260807 03:01:25.278183 31414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:25.280738 31427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "877103b851ca4626a1e02bab79b56a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877103b851ca4626a1e02bab79b56a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43873 } } }
I20260807 03:01:25.281234 31427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:25.280859 31428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 877103b851ca4626a1e02bab79b56a0b. Latest consensus state: current_term: 1 leader_uuid: "877103b851ca4626a1e02bab79b56a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "877103b851ca4626a1e02bab79b56a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43873 } } }
I20260807 03:01:25.285203 31428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:25.285643 31436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:25.291098 31436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:25.303026 31436 catalog_manager.cc:1347] Generated new cluster ID: a605b4f9e1ba4045b0179f68c7a8a1bc
I20260807 03:01:25.303243 31436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:25.320310 31436 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:25.320749 31436 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:25.326869 31436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 877103b851ca4626a1e02bab79b56a0b: Generated new TSK 0
I20260807 03:01:25.327648 31436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:25.341068 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31321
W20260807 03:01:25.342252 31407 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43539
W20260807 03:01:25.342993 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31321: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:25.343143 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 31321
2026-08-07T03:01:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:25 dist-test-slave-3h5h krb5kdc[31451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:25 dist-test-slave-3h5h krb5kdc[31451](info): set up 2 sockets
Aug 07 03:01:25 dist-test-slave-3h5h krb5kdc[31451](info): commencing operation
krb5kdc: starting...
W20260807 03:01:26.681509 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.003s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:26 dist-test-slave-3h5h krb5kdc[31451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:27.046283 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:27.408411 31467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:27.408769 31467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:27.408954 31467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:27.416231 31467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:27.416393 31467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:27.416500 31467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:27.416621 31467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:27.416704 31467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:27.426844 31467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:27.428395 31467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:27.429775 31467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:27.442739 31478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:27.442654 31481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:27.442786 31479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:27.446429 31467 server_base.cc:1034] running on GCE node
I20260807 03:01:27.447295 31467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:27.448800 31467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:27.450088 31467 hybrid_clock.cc:648] HybridClock initialized: now 1786071687450035 us; error 46 us; skew 500 ppm
I20260807 03:01:27.450472 31467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:27 dist-test-slave-3h5h krb5kdc[31451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:27.453778 31467 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:27.455394 31467 webserver.cc:460] Webserver started at http://127.0.0.1:35487/ using document root <none> and password file <none>
I20260807 03:01:27.456202 31467 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:27.456348 31467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:27.456663 31467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:27.459851 31467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51"
format_stamp: "Formatted at 2026-08-07 03:01:27 on dist-test-slave-3h5h"
server_key: "1bb84f4033b7ff9e835a419bbccaa826"
server_key_iv: "11096decbbc617121c2c4df083383f53"
server_key_version: "encryptionkey@0"
I20260807 03:01:27.460647 31467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51"
format_stamp: "Formatted at 2026-08-07 03:01:27 on dist-test-slave-3h5h"
server_key: "1bb84f4033b7ff9e835a419bbccaa826"
server_key_iv: "11096decbbc617121c2c4df083383f53"
server_key_version: "encryptionkey@0"
I20260807 03:01:27.465894 31467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 03:01:27.469404 31489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:27.470688 31467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:01:27.470875 31467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51"
format_stamp: "Formatted at 2026-08-07 03:01:27 on dist-test-slave-3h5h"
server_key: "1bb84f4033b7ff9e835a419bbccaa826"
server_key_iv: "11096decbbc617121c2c4df083383f53"
server_key_version: "encryptionkey@0"
I20260807 03:01:27.471073 31467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:27.486737 31467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:27.491335 31467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:27.491643 31467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:27.510807 31467 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34435
I20260807 03:01:27.510838 31557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34435 every 8 connection(s)
I20260807 03:01:27.512389 31467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:27.516362 31559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:27.521718 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 31467
I20260807 03:01:27.521924 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:27.522408 1509 external_mini_cluster.cc:1428] Setting key 3192656a199dd5b4a9706bb196e0820c
I20260807 03:01:27.527704 31559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51: Bootstrap starting.
I20260807 03:01:27.531561 31559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:27.532855 31559 log.cc:824] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:27.536471 31559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51: No bootstrap required, opened a new log
Aug 07 03:01:27 dist-test-slave-3h5h krb5kdc[31451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 03:01:27.544479 31559 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34435 } }
I20260807 03:01:27.544893 31559 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:27.545004 31559 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e7a1d4bfcd4759ab51f18dd960ec51, State: Initialized, Role: FOLLOWER
I20260807 03:01:27.545682 31559 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34435 } }
I20260807 03:01:27.545939 31559 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:27.546159 31559 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:27.546365 31559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:27.547312 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:27.524444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:34683 (local address 127.0.0.1:34435)
0807 03:01:27.525415 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:27.525426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:27.525475 (+ 49us) server_negotiation.cc:407] Connection header received
0807 03:01:27.527315 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:27.527340 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:27.527733 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:27.528098 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:27.530003 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:27.531307 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:27.533162 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:27.534345 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:27.534489 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:27.539354 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:27.539362 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:27.539377 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:27.539416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:27.542760 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:27.543597 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:27.543604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:27.543610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:27.543801 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:27.544317 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:27.544321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:27.544323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:27.545008 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:27.545350 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:27.545850 (+ 500us) server_negotiation.cc:299] Negotiation successful
0807 03:01:27.546194 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":497,"threads_started":1}
I20260807 03:01:27.548910 31559 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34435 } }
I20260807 03:01:27.549399 31559 leader_election.cc:302] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1e7a1d4bfcd4759ab51f18dd960ec51; no voters:
I20260807 03:01:27.550231 31559 leader_election.cc:288] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:27.550343 31568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:27.551481 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:27.552827 31559 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:27.557057 31568 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [term 1 LEADER]: Becoming Leader. State: Replica: c1e7a1d4bfcd4759ab51f18dd960ec51, State: Running, Role: LEADER
I20260807 03:01:27.557763 31568 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34435 } }
I20260807 03:01:27.561645 31569 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34435 } } }
I20260807 03:01:27.562199 31569 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:27.561919 31570 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e7a1d4bfcd4759ab51f18dd960ec51. Latest consensus state: current_term: 1 leader_uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e7a1d4bfcd4759ab51f18dd960ec51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34435 } } }
I20260807 03:01:27.562362 31570 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:27.563328 31586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:27.576550 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:27.556104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:33005 (local address 127.0.0.1:34435)
0807 03:01:27.556280 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:27.556286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:27.556921 (+ 635us) server_negotiation.cc:407] Connection header received
0807 03:01:27.557122 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:27.557127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:27.557237 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:27.557335 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:27.558830 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:27.559682 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:27.561629 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:27.562426 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:27.562599 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:27.566680 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:27.566690 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:27.566698 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:27.566755 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:27.571710 (+ 4955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:27.573147 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:27.573157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:27.573164 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:27.573375 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:27.574121 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:27.574130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:27.574136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:27.574683 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:27.574966 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:27.576147 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0807 03:01:27.576297 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 03:01:27.578682 31586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:27.589427 31586 catalog_manager.cc:1347] Generated new cluster ID: 0de06f11467a4279b1f1478a8e1b96ac
I20260807 03:01:27.589605 31586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:27.604574 31586 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:27.605933 31586 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:27.621774 31586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1e7a1d4bfcd4759ab51f18dd960ec51: Generated new TSK 0
I20260807 03:01:27.622699 31586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:27.633483 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31467
I20260807 03:01:27.635315 31552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:27.727592 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 31467
W20260807 03:01:27.753342 31449 connection.cc:535] client connection to 127.0.0.1:34435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34435): Connection reset by peer (error 104)
2026-08-07T03:01:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:27 dist-test-slave-3h5h krb5kdc[31604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:27 dist-test-slave-3h5h krb5kdc[31604](info): set up 2 sockets
Aug 07 03:01:27 dist-test-slave-3h5h krb5kdc[31604](info): commencing operation
krb5kdc: starting...
W20260807 03:01:29.108211 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.008s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:29 dist-test-slave-3h5h krb5kdc[31604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:29.488271 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:29.843262 31623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:29.843588 31623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:29.843755 31623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:29.851506 31623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:29.851749 31623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:29.851873 31623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:29.851994 31623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:29.852083 31623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:29.862686 31623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:29.864565 31623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:29.866618 31623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:29.879329 31631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:29.879477 31628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:29.879293 31629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:29.880927 31623 server_base.cc:1034] running on GCE node
I20260807 03:01:29.881834 31623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:29.883375 31623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:29.884577 31623 hybrid_clock.cc:648] HybridClock initialized: now 1786071689884519 us; error 54 us; skew 500 ppm
I20260807 03:01:29.884960 31623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:29 dist-test-slave-3h5h krb5kdc[31604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:29.888253 31623 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:29.889809 31623 webserver.cc:460] Webserver started at http://127.0.0.1:43719/ using document root <none> and password file <none>
I20260807 03:01:29.890522 31623 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:29.890647 31623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:29.890987 31623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:29.894299 31623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "4bc0e3f263f7492da42f1f9293f10742"
format_stamp: "Formatted at 2026-08-07 03:01:29 on dist-test-slave-3h5h"
server_key: "b52dceb4f8731498a4ee5c0e65cc4c5d"
server_key_iv: "2b451b2d60810ad8bd82c4bef8bfe0ef"
server_key_version: "encryptionkey@0"
I20260807 03:01:29.895102 31623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "4bc0e3f263f7492da42f1f9293f10742"
format_stamp: "Formatted at 2026-08-07 03:01:29 on dist-test-slave-3h5h"
server_key: "b52dceb4f8731498a4ee5c0e65cc4c5d"
server_key_iv: "2b451b2d60810ad8bd82c4bef8bfe0ef"
server_key_version: "encryptionkey@0"
I20260807 03:01:29.900483 31623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:01:29.904150 31638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:29.905588 31623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:01:29.905784 31623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "4bc0e3f263f7492da42f1f9293f10742"
format_stamp: "Formatted at 2026-08-07 03:01:29 on dist-test-slave-3h5h"
server_key: "b52dceb4f8731498a4ee5c0e65cc4c5d"
server_key_iv: "2b451b2d60810ad8bd82c4bef8bfe0ef"
server_key_version: "encryptionkey@0"
I20260807 03:01:29.905988 31623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:29.934280 31623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:29.938871 31623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:29.939188 31623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:29.958228 31623 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46677
I20260807 03:01:29.958248 31711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46677 every 8 connection(s)
I20260807 03:01:29.959777 31623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:29.962894 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 31623
I20260807 03:01:29.963100 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:29.963655 1509 external_mini_cluster.cc:1428] Setting key 9f07e49ed2593eb28ec476244fe66677
I20260807 03:01:29.964308 31713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:29.978773 31713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742: Bootstrap starting.
I20260807 03:01:29.978909 31718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:29.965710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:58387 (local address 127.0.0.1:46677)
0807 03:01:29.966534 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:29.966546 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:29.966590 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:01:29.969160 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:29.969197 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:29.969635 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:29.970018 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:29.970667 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:29.972129 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:29.974122 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:29.974823 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:29.974962 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:29.976383 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:29.976391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:29.976410 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:29.977073 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:29.977159 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:29.977196 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:29.977328 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:29.977617 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 03:01:29.977826 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":365,"threads_started":1}
I20260807 03:01:29.982266 31713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:29.982955 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:29.983572 31713 log.cc:824] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:29.987151 31713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742: No bootstrap required, opened a new log
I20260807 03:01:29.995131 31718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:29.988144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:59477 (local address 127.0.0.1:46677)
0807 03:01:29.988326 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:29.988332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:29.988866 (+ 534us) server_negotiation.cc:407] Connection header received
0807 03:01:29.989053 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:29.989059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:29.989153 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:29.989282 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:29.989773 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:29.990978 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:29.992320 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:29.993068 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:29.993222 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:29.994295 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:29.994301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:29.994307 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:29.994330 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:29.994391 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:29.994411 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:29.994513 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:29.994701 (+ 188us) server_negotiation.cc:299] Negotiation successful
0807 03:01:29.994762 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 03:01:29.996989 31713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc0e3f263f7492da42f1f9293f10742" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46677 } }
I20260807 03:01:29.997402 31713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:29.997519 31713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc0e3f263f7492da42f1f9293f10742, State: Initialized, Role: FOLLOWER
I20260807 03:01:29.998114 31713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc0e3f263f7492da42f1f9293f10742" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46677 } }
I20260807 03:01:29.998363 31713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:29.998526 31713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:29.998708 31713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:30.001024 31713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc0e3f263f7492da42f1f9293f10742" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46677 } }
I20260807 03:01:30.001546 31713 leader_election.cc:302] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bc0e3f263f7492da42f1f9293f10742; no voters:
I20260807 03:01:30.002195 31713 leader_election.cc:288] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:30.002401 31726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:30.003157 31726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc0e3f263f7492da42f1f9293f10742, State: Running, Role: LEADER
I20260807 03:01:30.003895 31726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc0e3f263f7492da42f1f9293f10742" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46677 } }
I20260807 03:01:30.005391 31713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:30.006963 31728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bc0e3f263f7492da42f1f9293f10742" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc0e3f263f7492da42f1f9293f10742" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46677 } } }
I20260807 03:01:30.007395 31728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:30.008343 31727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bc0e3f263f7492da42f1f9293f10742. Latest consensus state: current_term: 1 leader_uuid: "4bc0e3f263f7492da42f1f9293f10742" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc0e3f263f7492da42f1f9293f10742" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46677 } } }
I20260807 03:01:30.008762 31727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:30.011099 31732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:30.017710 31732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:30.027935 31732 catalog_manager.cc:1347] Generated new cluster ID: cd77033320074aff991f40a9de9aa936
I20260807 03:01:30.028090 31732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:30.044812 31732 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:30.047022 31732 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:30.059688 31732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bc0e3f263f7492da42f1f9293f10742: Generated new TSK 0
I20260807 03:01:30.061077 31732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:30.069296 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31623
W20260807 03:01:30.070791 31707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:58387
W20260807 03:01:30.071637 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31623: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:30.071827 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 31623
W20260807 03:01:30.100150 31602 connection.cc:535] client connection to 127.0.0.1:46677 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46677): Connection reset by peer (error 104)
2026-08-07T03:01:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:30 dist-test-slave-3h5h krb5kdc[31751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:30 dist-test-slave-3h5h krb5kdc[31751](info): set up 2 sockets
Aug 07 03:01:30 dist-test-slave-3h5h krb5kdc[31751](info): commencing operation
krb5kdc: starting...
W20260807 03:01:31.403918 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.284s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:31 dist-test-slave-3h5h krb5kdc[31751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:31.760082 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:32.122191 31768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:32.122483 31768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:32.122629 31768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:32.130136 31768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:32.130254 31768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:32.130360 31768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:32.130476 31768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:32.130546 31768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:32.140702 31768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:32.142103 31768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:32.143469 31768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:32.155797 31777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:32.155905 31775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:32.155944 31774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:32.157109 31768 server_base.cc:1034] running on GCE node
I20260807 03:01:32.157974 31768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:32.159520 31768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:32.160773 31768 hybrid_clock.cc:648] HybridClock initialized: now 1786071692160717 us; error 52 us; skew 500 ppm
I20260807 03:01:32.161188 31768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:32 dist-test-slave-3h5h krb5kdc[31751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:32.169900 31768 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:32.171465 31768 webserver.cc:460] Webserver started at http://127.0.0.1:35153/ using document root <none> and password file <none>
I20260807 03:01:32.172154 31768 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:32.172266 31768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:32.172559 31768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:32.175829 31768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "cb796f1002bf401cbff9b31c8c80d3b3"
format_stamp: "Formatted at 2026-08-07 03:01:32 on dist-test-slave-3h5h"
server_key: "c9577aca11a5bbf153a7c2392d620e26"
server_key_iv: "da04f6f4a9cf502e5cd315c6e3af32f1"
server_key_version: "encryptionkey@0"
I20260807 03:01:32.176578 31768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "cb796f1002bf401cbff9b31c8c80d3b3"
format_stamp: "Formatted at 2026-08-07 03:01:32 on dist-test-slave-3h5h"
server_key: "c9577aca11a5bbf153a7c2392d620e26"
server_key_iv: "da04f6f4a9cf502e5cd315c6e3af32f1"
server_key_version: "encryptionkey@0"
I20260807 03:01:32.182204 31768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 03:01:32.185791 31784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:32.187140 31768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:01:32.187325 31768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "cb796f1002bf401cbff9b31c8c80d3b3"
format_stamp: "Formatted at 2026-08-07 03:01:32 on dist-test-slave-3h5h"
server_key: "c9577aca11a5bbf153a7c2392d620e26"
server_key_iv: "da04f6f4a9cf502e5cd315c6e3af32f1"
server_key_version: "encryptionkey@0"
I20260807 03:01:32.187502 31768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:32.193519 31768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:32.198045 31768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:32.198362 31768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:32.217048 31857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43109 every 8 connection(s)
I20260807 03:01:32.217176 31768 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43109
I20260807 03:01:32.218716 31768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:32.222543 31858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:32.225310 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 31768
I20260807 03:01:32.225502 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:32.225972 1509 external_mini_cluster.cc:1428] Setting key e37d50e03b8f91db798de8130748240c
I20260807 03:01:32.235317 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:32.227977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:42893 (local address 127.0.0.1:43109)
0807 03:01:32.228877 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:32.228894 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:32.228945 (+ 51us) server_negotiation.cc:407] Connection header received
0807 03:01:32.231303 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:32.231332 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:32.231766 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:32.232225 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:32.233280 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:32.233289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:32.233306 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:32.233326 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:32.233397 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:32.233430 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:32.233663 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:32.233832 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 03:01:32.234057 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":354,"threads_started":1}
I20260807 03:01:32.236362 31858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3: Bootstrap starting.
I20260807 03:01:32.239153 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:32.239878 31858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:32.241206 31858 log.cc:824] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:32.244697 31858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3: No bootstrap required, opened a new log
I20260807 03:01:32.247056 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:32.244433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:48019 (local address 127.0.0.1:43109)
0807 03:01:32.244671 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:32.244686 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:32.245062 (+ 376us) server_negotiation.cc:407] Connection header received
0807 03:01:32.245258 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:32.245262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:32.245309 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:32.245393 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:32.246372 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:32.246376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:32.246379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:32.246394 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:32.246432 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:32.246446 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:32.246531 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:32.246692 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 03:01:32.246747 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:01:32.252789 31858 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb796f1002bf401cbff9b31c8c80d3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } }
I20260807 03:01:32.253233 31858 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:32.253347 31858 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb796f1002bf401cbff9b31c8c80d3b3, State: Initialized, Role: FOLLOWER
I20260807 03:01:32.253942 31858 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb796f1002bf401cbff9b31c8c80d3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } }
I20260807 03:01:32.254225 31858 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:32.254377 31858 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:32.254565 31858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:32.256736 31858 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb796f1002bf401cbff9b31c8c80d3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } }
I20260807 03:01:32.257274 31858 leader_election.cc:302] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb796f1002bf401cbff9b31c8c80d3b3; no voters:
I20260807 03:01:32.257853 31858 leader_election.cc:288] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:32.258047 31870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:32.258731 31870 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [term 1 LEADER]: Becoming Leader. State: Replica: cb796f1002bf401cbff9b31c8c80d3b3, State: Running, Role: LEADER
I20260807 03:01:32.259326 31870 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb796f1002bf401cbff9b31c8c80d3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } }
I20260807 03:01:32.259996 31858 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:32.262298 31871 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb796f1002bf401cbff9b31c8c80d3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb796f1002bf401cbff9b31c8c80d3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } } }
I20260807 03:01:32.262312 31874 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb796f1002bf401cbff9b31c8c80d3b3. Latest consensus state: current_term: 1 leader_uuid: "cb796f1002bf401cbff9b31c8c80d3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb796f1002bf401cbff9b31c8c80d3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } } }
I20260807 03:01:32.262945 31871 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:32.262969 31874 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:32.263989 31880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:32.269490 31880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:32.279315 31880 catalog_manager.cc:1347] Generated new cluster ID: b3072261545042e4b15f5a9ec695f69e
I20260807 03:01:32.279455 31880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:32.309222 31880 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:32.309782 31880 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:32.320386 31880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb796f1002bf401cbff9b31c8c80d3b3: Generated new TSK 0
I20260807 03:01:32.321286 31880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:32.334241 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31768
W20260807 03:01:32.335395 31853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:42893
W20260807 03:01:32.336150 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31768: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:32.336309 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 31768
2026-08-07T03:01:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2262 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:32 dist-test-slave-3h5h krb5kdc[31898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:32 dist-test-slave-3h5h krb5kdc[31898](info): set up 2 sockets
Aug 07 03:01:32 dist-test-slave-3h5h krb5kdc[31898](info): commencing operation
krb5kdc: starting...
W20260807 03:01:33.702090 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.320s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:33 dist-test-slave-3h5h krb5kdc[31898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:34.070505 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:34.432571 31920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:34.432965 31920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:34.433125 31920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:34.440627 31920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:34.440747 31920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:34.440794 31920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:34.440907 31920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:34.440979 31920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:34.451258 31920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:34.452606 31920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:34.453959 31920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:34.466111 31926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:34.466140 31930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:34.466065 31927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:34.467187 31920 server_base.cc:1034] running on GCE node
I20260807 03:01:34.468012 31920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:34.469554 31920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:34.470783 31920 hybrid_clock.cc:648] HybridClock initialized: now 1786071694470754 us; error 58 us; skew 500 ppm
I20260807 03:01:34.471154 31920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:34 dist-test-slave-3h5h krb5kdc[31898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:34.474458 31920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:34.476006 31920 webserver.cc:460] Webserver started at http://127.0.0.1:40517/ using document root <none> and password file <none>
I20260807 03:01:34.476794 31920 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:34.476956 31920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:34.477233 31920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:34.480188 31920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "cd42e12903f8481982fc00a551fc636f"
format_stamp: "Formatted at 2026-08-07 03:01:34 on dist-test-slave-3h5h"
server_key: "2b5dabbd268327d3904374eb251c41de"
server_key_iv: "c502798d08cd49fadc0026744c7cd83d"
server_key_version: "encryptionkey@0"
I20260807 03:01:34.481029 31920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "cd42e12903f8481982fc00a551fc636f"
format_stamp: "Formatted at 2026-08-07 03:01:34 on dist-test-slave-3h5h"
server_key: "2b5dabbd268327d3904374eb251c41de"
server_key_iv: "c502798d08cd49fadc0026744c7cd83d"
server_key_version: "encryptionkey@0"
I20260807 03:01:34.486349 31920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:01:34.489770 31939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:34.491061 31920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:01:34.491250 31920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "cd42e12903f8481982fc00a551fc636f"
format_stamp: "Formatted at 2026-08-07 03:01:34 on dist-test-slave-3h5h"
server_key: "2b5dabbd268327d3904374eb251c41de"
server_key_iv: "c502798d08cd49fadc0026744c7cd83d"
server_key_version: "encryptionkey@0"
I20260807 03:01:34.491442 31920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:34.530908 31920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:34.535943 31920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:34.536257 31920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:34.554646 31920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45283
I20260807 03:01:34.554687 32008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45283 every 8 connection(s)
I20260807 03:01:34.556063 31920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:34.557969 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 31920
I20260807 03:01:34.558161 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:34.558645 1509 external_mini_cluster.cc:1428] Setting key 017781970ca90df9ba695ec10f366bf4
I20260807 03:01:34.560482 32010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 03:01:34 dist-test-slave-3h5h krb5kdc[31898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 03:01:34.574527 32010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f: Bootstrap starting.
I20260807 03:01:34.578917 32010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:34.580240 32010 log.cc:824] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:34.583870 32010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f: No bootstrap required, opened a new log
I20260807 03:01:34.584095 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:34.560436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:42509 (local address 127.0.0.1:45283)
0807 03:01:34.561355 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:34.561371 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:34.561422 (+ 51us) server_negotiation.cc:407] Connection header received
0807 03:01:34.563643 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:34.563670 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:34.564221 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:34.564677 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:34.566053 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:34.568110 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:34.569568 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:34.570647 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:34.570853 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:34.575230 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:34.575242 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:34.575262 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:34.575315 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:34.579466 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:34.580290 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:34.580300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:34.580308 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:34.580520 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:34.581096 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:34.581100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:34.581102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:34.581653 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:34.582017 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:34.582416 (+ 399us) server_negotiation.cc:299] Negotiation successful
0807 03:01:34.582767 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":434,"threads_started":1}
I20260807 03:01:34.588037 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:34.593994 32010 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42e12903f8481982fc00a551fc636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45283 } }
I20260807 03:01:34.594404 32010 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:34.594537 32010 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd42e12903f8481982fc00a551fc636f, State: Initialized, Role: FOLLOWER
I20260807 03:01:34.595309 32010 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42e12903f8481982fc00a551fc636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45283 } }
I20260807 03:01:34.595546 32010 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:34.595700 32010 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:34.595881 32010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:34.598825 32010 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42e12903f8481982fc00a551fc636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45283 } }
I20260807 03:01:34.599406 32010 leader_election.cc:302] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd42e12903f8481982fc00a551fc636f; no voters:
I20260807 03:01:34.600155 32010 leader_election.cc:288] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:34.600296 32023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:34.602388 32010 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:34.607379 32023 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [term 1 LEADER]: Becoming Leader. State: Replica: cd42e12903f8481982fc00a551fc636f, State: Running, Role: LEADER
I20260807 03:01:34.608119 32023 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42e12903f8481982fc00a551fc636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45283 } }
I20260807 03:01:34.613904 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:34.593668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:43511 (local address 127.0.0.1:45283)
0807 03:01:34.593879 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:34.593885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:34.594334 (+ 449us) server_negotiation.cc:407] Connection header received
0807 03:01:34.594574 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:34.594579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:34.594668 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:34.594812 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:34.596632 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:34.597860 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:34.599157 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:34.599875 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:34.600041 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:34.604281 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:34.604291 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:34.604300 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:34.604363 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:34.608870 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:34.609900 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:34.609910 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:34.609916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:34.610125 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:34.610890 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:34.610899 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:34.610904 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:34.611468 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:34.612027 (+ 559us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:34.613000 (+ 973us) server_negotiation.cc:299] Negotiation successful
0807 03:01:34.613300 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 03:01:34.618881 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd42e12903f8481982fc00a551fc636f. Latest consensus state: current_term: 1 leader_uuid: "cd42e12903f8481982fc00a551fc636f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42e12903f8481982fc00a551fc636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45283 } } }
I20260807 03:01:34.619203 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:34.618387 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd42e12903f8481982fc00a551fc636f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42e12903f8481982fc00a551fc636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45283 } } }
I20260807 03:01:34.619988 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f [sys.catalog]: This master's current role is: LEADER
W20260807 03:01:34.622841 32045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:01:34.622957 32045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:01:34.623106 32048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:34.625370 32048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:34.634753 32048 catalog_manager.cc:1347] Generated new cluster ID: 26a7b3ea05054fa88c68e300df130de4
I20260807 03:01:34.634876 32048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:34.649870 32048 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:34.651782 32048 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:34.675395 32048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd42e12903f8481982fc00a551fc636f: Generated new TSK 0
I20260807 03:01:34.676532 32048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:34.689248 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 31920
I20260807 03:01:34.690433 32001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:34.784196 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 31920
W20260807 03:01:34.811058 31896 connection.cc:535] client connection to 127.0.0.1:45283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45283): Connection reset by peer (error 104)
2026-08-07T03:01:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:34 dist-test-slave-3h5h krb5kdc[32057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:34 dist-test-slave-3h5h krb5kdc[32057](info): set up 2 sockets
Aug 07 03:01:34 dist-test-slave-3h5h krb5kdc[32057](info): commencing operation
krb5kdc: starting...
W20260807 03:01:36.230459 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.400s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:36 dist-test-slave-3h5h krb5kdc[32057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:36.596061 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:36.966509 32077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:36.966853 32077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:36.966967 32077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:36.974752 32077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:36.974900 32077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:36.974977 32077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:36.975054 32077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:36.975108 32077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:36.985649 32077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:36.987089 32077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:36.988507 32077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:37.001745 32085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:37.001812 32087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:37.001842 32084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:37.002568 32077 server_base.cc:1034] running on GCE node
I20260807 03:01:37.003402 32077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:37.005035 32077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:37.006289 32077 hybrid_clock.cc:648] HybridClock initialized: now 1786071697006247 us; error 38 us; skew 500 ppm
I20260807 03:01:37.006700 32077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:37 dist-test-slave-3h5h krb5kdc[32057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:37.010015 32077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:37.011531 32077 webserver.cc:460] Webserver started at http://127.0.0.1:42463/ using document root <none> and password file <none>
I20260807 03:01:37.012316 32077 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:37.012470 32077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:37.012789 32077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:37.016170 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "00834bdf4adf4731a635bb1da8e282b4"
format_stamp: "Formatted at 2026-08-07 03:01:37 on dist-test-slave-3h5h"
server_key: "f81fcbf4cafd3f410407dc48cfe9cc63"
server_key_iv: "5a42cd71543d7cd355133e71611fbee0"
server_key_version: "encryptionkey@0"
I20260807 03:01:37.017086 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "00834bdf4adf4731a635bb1da8e282b4"
format_stamp: "Formatted at 2026-08-07 03:01:37 on dist-test-slave-3h5h"
server_key: "f81fcbf4cafd3f410407dc48cfe9cc63"
server_key_iv: "5a42cd71543d7cd355133e71611fbee0"
server_key_version: "encryptionkey@0"
I20260807 03:01:37.022461 32077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:01:37.026381 32097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:37.028137 32077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:01:37.028345 32077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "00834bdf4adf4731a635bb1da8e282b4"
format_stamp: "Formatted at 2026-08-07 03:01:37 on dist-test-slave-3h5h"
server_key: "f81fcbf4cafd3f410407dc48cfe9cc63"
server_key_iv: "5a42cd71543d7cd355133e71611fbee0"
server_key_version: "encryptionkey@0"
I20260807 03:01:37.028592 32077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:37.041709 32077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:37.046700 32077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:37.046976 32077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:37.068037 32077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38115
I20260807 03:01:37.068085 32164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38115 every 8 connection(s)
I20260807 03:01:37.070123 32077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:37.074461 32166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:37.076270 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 32077
I20260807 03:01:37.076480 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:37.077008 1509 external_mini_cluster.cc:1428] Setting key d235e1dee0d7156b2e2df662e5c3e649
I20260807 03:01:37.089170 32166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4: Bootstrap starting.
I20260807 03:01:37.091070 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:37.078889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:35433 (local address 127.0.0.1:38115)
0807 03:01:37.079772 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:37.079784 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:37.079829 (+ 45us) server_negotiation.cc:407] Connection header received
0807 03:01:37.081574 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:37.081605 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:37.082032 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:37.082409 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:37.083076 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:37.084336 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:37.086318 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:37.087037 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:37.087195 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:37.088555 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:37.088563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:37.088578 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:37.089225 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:37.089315 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:37.089375 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:37.089519 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:37.089824 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 03:01:37.090042 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":403,"threads_started":1}
I20260807 03:01:37.093132 32166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:37.094563 32166 log.cc:824] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:37.095446 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:37.097711 32166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4: No bootstrap required, opened a new log
I20260807 03:01:37.106793 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:37.100222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:55377 (local address 127.0.0.1:38115)
0807 03:01:37.100426 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:37.100433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:37.100826 (+ 393us) server_negotiation.cc:407] Connection header received
0807 03:01:37.101078 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:37.101082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:37.101149 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:37.101268 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:37.101840 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:37.102623 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:37.103977 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:37.104449 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:37.104624 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:37.105996 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:37.106001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:37.106005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:37.106023 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:37.106065 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:37.106080 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:37.106211 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:37.106444 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 03:01:37.106502 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:01:37.106426 32166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00834bdf4adf4731a635bb1da8e282b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } }
I20260807 03:01:37.107242 32166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:37.107393 32166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00834bdf4adf4731a635bb1da8e282b4, State: Initialized, Role: FOLLOWER
I20260807 03:01:37.108196 32166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00834bdf4adf4731a635bb1da8e282b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } }
I20260807 03:01:37.108530 32166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:37.108709 32166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:37.108950 32166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:37.111807 32166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00834bdf4adf4731a635bb1da8e282b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } }
I20260807 03:01:37.112407 32166 leader_election.cc:302] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00834bdf4adf4731a635bb1da8e282b4; no voters:
I20260807 03:01:37.113121 32166 leader_election.cc:288] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:37.113265 32176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:37.114141 32176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [term 1 LEADER]: Becoming Leader. State: Replica: 00834bdf4adf4731a635bb1da8e282b4, State: Running, Role: LEADER
I20260807 03:01:37.114714 32176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00834bdf4adf4731a635bb1da8e282b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } }
I20260807 03:01:37.115685 32166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:37.118160 32177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00834bdf4adf4731a635bb1da8e282b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00834bdf4adf4731a635bb1da8e282b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } } }
I20260807 03:01:37.118151 32178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00834bdf4adf4731a635bb1da8e282b4. Latest consensus state: current_term: 1 leader_uuid: "00834bdf4adf4731a635bb1da8e282b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00834bdf4adf4731a635bb1da8e282b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } } }
I20260807 03:01:37.118607 32178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:37.118608 32177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:37.122014 32183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:37.127965 32183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:37.138746 32183 catalog_manager.cc:1347] Generated new cluster ID: 92a30cc16d6d4e63a37a052a2ddf7eff
I20260807 03:01:37.139021 32183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:37.149807 32183 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:37.151212 32183 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:37.160730 32183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00834bdf4adf4731a635bb1da8e282b4: Generated new TSK 0
I20260807 03:01:37.161892 32183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:37.181223 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 32077
W20260807 03:01:37.182535 32160 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:35433
W20260807 03:01:37.183302 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 32077: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:37.183447 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 32077
W20260807 03:01:37.210479 32053 connection.cc:535] client connection to 127.0.0.1:38115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38115): Connection reset by peer (error 104)
2026-08-07T03:01:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:37 dist-test-slave-3h5h krb5kdc[32206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:37 dist-test-slave-3h5h krb5kdc[32206](info): set up 2 sockets
Aug 07 03:01:37 dist-test-slave-3h5h krb5kdc[32206](info): commencing operation
krb5kdc: starting...
W20260807 03:01:38.530506 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.300s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:38 dist-test-slave-3h5h krb5kdc[32206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:01:38.889062 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:39.254199 32234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:39.254506 32234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:39.254601 32234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:39.262025 32234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:39.262159 32234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:39.262223 32234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:39.262287 32234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:39.262332 32234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:39.272323 32234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:39.273813 32234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:39.275103 32234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:39.287125 32242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:39.287165 32241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:39.287209 32244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:39.287941 32234 server_base.cc:1034] running on GCE node
I20260807 03:01:39.288733 32234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:39.290198 32234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:39.291443 32234 hybrid_clock.cc:648] HybridClock initialized: now 1786071699291395 us; error 35 us; skew 500 ppm
I20260807 03:01:39.291810 32234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:39 dist-test-slave-3h5h krb5kdc[32206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:39.298878 32234 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:01:39.302145 32234 webserver.cc:460] Webserver started at http://127.0.0.1:44591/ using document root <none> and password file <none>
I20260807 03:01:39.303211 32234 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:39.303366 32234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:39.303761 32234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:39.307780 32234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "71ba582f5269499bb38ff80f2412317f"
format_stamp: "Formatted at 2026-08-07 03:01:39 on dist-test-slave-3h5h"
server_key: "56b8ad073d37e1f96c6177751e9304aa"
server_key_iv: "700cae1f593823428589b3c0f3e82ec6"
server_key_version: "encryptionkey@0"
I20260807 03:01:39.308565 32234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "71ba582f5269499bb38ff80f2412317f"
format_stamp: "Formatted at 2026-08-07 03:01:39 on dist-test-slave-3h5h"
server_key: "56b8ad073d37e1f96c6177751e9304aa"
server_key_iv: "700cae1f593823428589b3c0f3e82ec6"
server_key_version: "encryptionkey@0"
I20260807 03:01:39.313905 32234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:01:39.317404 32256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:39.318773 32234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:01:39.318962 32234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "71ba582f5269499bb38ff80f2412317f"
format_stamp: "Formatted at 2026-08-07 03:01:39 on dist-test-slave-3h5h"
server_key: "56b8ad073d37e1f96c6177751e9304aa"
server_key_iv: "700cae1f593823428589b3c0f3e82ec6"
server_key_version: "encryptionkey@0"
I20260807 03:01:39.319162 32234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:39.324960 32234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:39.329356 32234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:39.329669 32234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:39.349426 32234 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39633
I20260807 03:01:39.349469 32328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39633 every 8 connection(s)
I20260807 03:01:39.350976 32234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:39.354851 32329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:39.355248 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 32234
I20260807 03:01:39.355494 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:39.355965 1509 external_mini_cluster.cc:1428] Setting key 7c92872d171dcbd3464b5d5f34b92e80
I20260807 03:01:39.364593 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:39.357900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:58397 (local address 127.0.0.1:39633)
0807 03:01:39.358855 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:39.358867 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:39.358917 (+ 50us) server_negotiation.cc:407] Connection header received
0807 03:01:39.361146 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:39.361170 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:39.361520 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:39.361863 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:39.362982 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:39.362988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:39.363002 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:39.363016 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:39.363077 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:39.363100 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:39.363269 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:39.363411 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 03:01:39.363619 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":506,"threads_started":1}
I20260807 03:01:39.368574 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:39.368952 32329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f: Bootstrap starting.
I20260807 03:01:39.372287 32329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:39.373819 32329 log.cc:824] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:39.375945 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:39.373195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:57701 (local address 127.0.0.1:39633)
0807 03:01:39.373383 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:39.373397 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:39.373899 (+ 502us) server_negotiation.cc:407] Connection header received
0807 03:01:39.374126 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:39.374131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:39.374193 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:39.374298 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:39.375275 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:39.375281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:39.375286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:39.375304 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:39.375352 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:39.375367 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:39.375488 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:39.375590 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 03:01:39.375667 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:01:39.376971 32329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f: No bootstrap required, opened a new log
I20260807 03:01:39.384814 32329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ba582f5269499bb38ff80f2412317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } }
I20260807 03:01:39.385304 32329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:39.385413 32329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ba582f5269499bb38ff80f2412317f, State: Initialized, Role: FOLLOWER
I20260807 03:01:39.386097 32329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ba582f5269499bb38ff80f2412317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } }
I20260807 03:01:39.386354 32329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:39.386507 32329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:39.386684 32329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:39.389973 32329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ba582f5269499bb38ff80f2412317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } }
I20260807 03:01:39.390620 32329 leader_election.cc:302] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71ba582f5269499bb38ff80f2412317f; no voters:
I20260807 03:01:39.391551 32329 leader_election.cc:288] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:39.391755 32341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:39.394310 32329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:39.396991 32341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [term 1 LEADER]: Becoming Leader. State: Replica: 71ba582f5269499bb38ff80f2412317f, State: Running, Role: LEADER
I20260807 03:01:39.398006 32341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ba582f5269499bb38ff80f2412317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } }
I20260807 03:01:39.406764 32343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ba582f5269499bb38ff80f2412317f. Latest consensus state: current_term: 1 leader_uuid: "71ba582f5269499bb38ff80f2412317f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ba582f5269499bb38ff80f2412317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } } }
I20260807 03:01:39.407364 32343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:39.408416 32342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71ba582f5269499bb38ff80f2412317f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ba582f5269499bb38ff80f2412317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } } }
I20260807 03:01:39.409338 32342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f [sys.catalog]: This master's current role is: LEADER
W20260807 03:01:39.419958 32358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:01:39.420248 32358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:01:39.420506 32357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:39.423138 32357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:39.434702 32357 catalog_manager.cc:1347] Generated new cluster ID: 66ca98c794f846c5b164b4dbd793c214
I20260807 03:01:39.434890 32357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:39.453549 32357 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:39.454064 32357 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:39.481264 32357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71ba582f5269499bb38ff80f2412317f: Generated new TSK 0
I20260807 03:01:39.482326 32357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:39.491055 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 32234
W20260807 03:01:39.492478 32323 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:58397
W20260807 03:01:39.493367 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 32234: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:39.493534 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 32234
2026-08-07T03:01:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:39 dist-test-slave-3h5h krb5kdc[32367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:39 dist-test-slave-3h5h krb5kdc[32367](info): set up 2 sockets
Aug 07 03:01:39 dist-test-slave-3h5h krb5kdc[32367](info): commencing operation
krb5kdc: starting...
W20260807 03:01:40.870831 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.333s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:41 dist-test-slave-3h5h krb5kdc[32367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:01:41.245127 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34161
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:34161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:41.609511 32382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:41.609815 32382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:41.609918 32382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:41.617360 32382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:41.617478 32382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:41.617537 32382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:41.617591 32382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:41.617637 32382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:41.627511 32382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:34161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:41.628849 32382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:41.630192 32382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:41.645311 32390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:41.645417 32394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:41.645345 32392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:41.646132 32382 server_base.cc:1034] running on GCE node
I20260807 03:01:41.646960 32382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:41.648442 32382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:41.649688 32382 hybrid_clock.cc:648] HybridClock initialized: now 1786071701649642 us; error 37 us; skew 500 ppm
I20260807 03:01:41.650056 32382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:41 dist-test-slave-3h5h krb5kdc[32367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071701, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:41.653478 32382 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:01:41.655097 32382 webserver.cc:460] Webserver started at http://127.1.121.126:46433/ using document root <none> and password file <none>
I20260807 03:01:41.655862 32382 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:41.656011 32382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:41.656354 32382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:41.659525 32382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "d4d44abc16074c9d8f5af52feb1fdbe8"
format_stamp: "Formatted at 2026-08-07 03:01:41 on dist-test-slave-3h5h"
server_key: "f850b92f1f9dc9e24309eebbfbbaa075"
server_key_iv: "97e20f1ed186594947275bb590b82b89"
server_key_version: "encryptionkey@0"
I20260807 03:01:41.660342 32382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "d4d44abc16074c9d8f5af52feb1fdbe8"
format_stamp: "Formatted at 2026-08-07 03:01:41 on dist-test-slave-3h5h"
server_key: "f850b92f1f9dc9e24309eebbfbbaa075"
server_key_iv: "97e20f1ed186594947275bb590b82b89"
server_key_version: "encryptionkey@0"
I20260807 03:01:41.665773 32382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 03:01:41.669384 32405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:41.670845 32382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:01:41.670994 32382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "d4d44abc16074c9d8f5af52feb1fdbe8"
format_stamp: "Formatted at 2026-08-07 03:01:41 on dist-test-slave-3h5h"
server_key: "f850b92f1f9dc9e24309eebbfbbaa075"
server_key_iv: "97e20f1ed186594947275bb590b82b89"
server_key_version: "encryptionkey@0"
I20260807 03:01:41.671160 32382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:41.684219 32382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:41.688685 32382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:41.688997 32382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:41.708194 32382 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:34161
I20260807 03:01:41.708214 32477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:34161 every 8 connection(s)
I20260807 03:01:41.709651 32382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:41.713485 32478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:41.719884 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 32382
I20260807 03:01:41.720093 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:41.720706 1509 external_mini_cluster.cc:1428] Setting key d27a930535b7e3c86923c491d1908a5f
I20260807 03:01:41.724623 32478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8: Bootstrap starting.
I20260807 03:01:41.728477 32478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:41.729655 32478 log.cc:824] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:41.732903 32478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8: No bootstrap required, opened a new log
Aug 07 03:01:41 dist-test-slave-3h5h krb5kdc[32367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:01:41.743894 32478 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d44abc16074c9d8f5af52feb1fdbe8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34161 } }
I20260807 03:01:41.744407 32478 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:41.744525 32478 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d44abc16074c9d8f5af52feb1fdbe8, State: Initialized, Role: FOLLOWER
I20260807 03:01:41.745231 32478 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d44abc16074c9d8f5af52feb1fdbe8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34161 } }
I20260807 03:01:41.745498 32478 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:41.745685 32478 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:41.745889 32478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:41.746227 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:41.722626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33745 (local address 127.1.121.126:34161)
0807 03:01:41.723422 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:41.723433 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:41.723475 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:01:41.725264 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:41.725286 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:41.725647 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:41.725965 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:41.727959 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:41.729399 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:41.731048 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:41.731714 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:41.731843 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:41.737017 (+ 5174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:41.737028 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:41.737051 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:41.737133 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:41.741271 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:41.742347 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:41.742354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:41.742361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:41.742553 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:41.743059 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:41.743062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:41.743064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:41.743605 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:41.743852 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:41.744464 (+ 612us) server_negotiation.cc:299] Negotiation successful
0807 03:01:41.744916 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":377,"threads_started":1}
I20260807 03:01:41.748474 32478 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d44abc16074c9d8f5af52feb1fdbe8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34161 } }
I20260807 03:01:41.749079 32478 leader_election.cc:302] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4d44abc16074c9d8f5af52feb1fdbe8; no voters:
I20260807 03:01:41.749756 32478 leader_election.cc:288] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:41.749895 32483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:41.750699 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:41.751394 32483 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [term 1 LEADER]: Becoming Leader. State: Replica: d4d44abc16074c9d8f5af52feb1fdbe8, State: Running, Role: LEADER
I20260807 03:01:41.752018 32478 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:41.751924 32483 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d44abc16074c9d8f5af52feb1fdbe8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34161 } }
I20260807 03:01:41.754639 32486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d44abc16074c9d8f5af52feb1fdbe8. Latest consensus state: current_term: 1 leader_uuid: "d4d44abc16074c9d8f5af52feb1fdbe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d44abc16074c9d8f5af52feb1fdbe8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34161 } } }
I20260807 03:01:41.755045 32486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:41.755613 32485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4d44abc16074c9d8f5af52feb1fdbe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d44abc16074c9d8f5af52feb1fdbe8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34161 } } }
I20260807 03:01:41.755964 32485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:41.777987 32491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:41.790373 32491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:41.809772 32491 catalog_manager.cc:1347] Generated new cluster ID: 07571914e1e14bfcbce77ad981d9eeac
I20260807 03:01:41.809922 32491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:41.824014 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:41.803437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53909 (local address 127.1.121.126:34161)
0807 03:01:41.803637 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:41.803645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:41.806028 (+ 2383us) server_negotiation.cc:407] Connection header received
0807 03:01:41.806322 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:41.806328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:41.806417 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:41.806533 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:41.808289 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:41.809542 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:41.811481 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:41.812162 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:41.812276 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:41.815907 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:41.815917 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:41.815926 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:41.815966 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:41.819900 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:41.821091 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:41.821099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:41.821103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:41.821316 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:41.821935 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:41.821941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:41.821944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:41.822383 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:41.822672 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:41.823414 (+ 742us) server_negotiation.cc:299] Negotiation successful
0807 03:01:41.823614 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:01:41.837321 32491 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:41.839203 32491 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:41.847091 32491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4d44abc16074c9d8f5af52feb1fdbe8: Generated new TSK 0
I20260807 03:01:41.848268 32491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:41.859441 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 32382
I20260807 03:01:41.860633 32473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:41.960569 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 32382
W20260807 03:01:41.988735 32362 connection.cc:535] client connection to 127.1.121.126:34161 recv error: Network error: failed to read from TLS socket (remote: 127.1.121.126:34161): Connection reset by peer (error 104)
2026-08-07T03:01:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:42 dist-test-slave-3h5h krb5kdc[32521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:42 dist-test-slave-3h5h krb5kdc[32521](info): set up 2 sockets
Aug 07 03:01:42 dist-test-slave-3h5h krb5kdc[32521](info): commencing operation
krb5kdc: starting...
W20260807 03:01:43.303242 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.001s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:43 dist-test-slave-3h5h krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:01:43.664304 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:46227
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:44.025162 32541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:44.025563 32541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:44.025738 32541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:44.033201 32541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:44.033392 32541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:44.033505 32541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:44.033620 32541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:44.033730 32541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:44.043956 32541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:46227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:46227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:44.045475 32541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:44.046941 32541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:44.059590 32548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:44.059628 32549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:44.059657 32551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:44.060456 32541 server_base.cc:1034] running on GCE node
I20260807 03:01:44.061349 32541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:44.062819 32541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:44.064014 32541 hybrid_clock.cc:648] HybridClock initialized: now 1786071704063950 us; error 54 us; skew 500 ppm
I20260807 03:01:44.064347 32541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:44 dist-test-slave-3h5h krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071704, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:44.072548 32541 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:01:44.074448 32541 webserver.cc:460] Webserver started at http://127.1.121.126:45053/ using document root <none> and password file <none>
I20260807 03:01:44.075201 32541 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:44.075349 32541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:44.075680 32541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:44.079030 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "97d34158b1f040a196c9ce63943aa11f"
format_stamp: "Formatted at 2026-08-07 03:01:44 on dist-test-slave-3h5h"
server_key: "dc2071e6290d0026681b33d179e491a4"
server_key_iv: "f44b92f030e806c53d5e20261e57b414"
server_key_version: "encryptionkey@0"
I20260807 03:01:44.079838 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "97d34158b1f040a196c9ce63943aa11f"
format_stamp: "Formatted at 2026-08-07 03:01:44 on dist-test-slave-3h5h"
server_key: "dc2071e6290d0026681b33d179e491a4"
server_key_iv: "f44b92f030e806c53d5e20261e57b414"
server_key_version: "encryptionkey@0"
I20260807 03:01:44.085309 32541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:01:44.089566 32560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:44.090963 32541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:01:44.091147 32541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "97d34158b1f040a196c9ce63943aa11f"
format_stamp: "Formatted at 2026-08-07 03:01:44 on dist-test-slave-3h5h"
server_key: "dc2071e6290d0026681b33d179e491a4"
server_key_iv: "f44b92f030e806c53d5e20261e57b414"
server_key_version: "encryptionkey@0"
I20260807 03:01:44.091363 32541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:44.115515 32541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:44.119913 32541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:44.120232 32541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:44.140118 32541 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:46227
I20260807 03:01:44.140151 32631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:46227 every 8 connection(s)
I20260807 03:01:44.141700 32541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:44.145711 32632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:44.150184 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 32541
I20260807 03:01:44.150385 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:44.150980 1509 external_mini_cluster.cc:1428] Setting key f60a5bcc03272a0c423119fb53cebb8e
I20260807 03:01:44.158303 32632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f: Bootstrap starting.
I20260807 03:01:44.162308 32632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:44.163736 32632 log.cc:824] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:44.166395 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:44.153051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33707 (local address 127.1.121.126:46227)
0807 03:01:44.153943 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:44.153955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:44.154001 (+ 46us) server_negotiation.cc:407] Connection header received
0807 03:01:44.156402 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:44.156429 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:44.156959 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:44.157332 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:44.158024 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:44.159473 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:44.161448 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:44.162134 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:44.162296 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:44.163676 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:44.163687 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:44.163704 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:44.164455 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:44.164581 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:44.164618 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:44.164813 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:44.165156 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 03:01:44.165366 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":423,"threads_started":1}
I20260807 03:01:44.167268 32632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f: No bootstrap required, opened a new log
I20260807 03:01:44.170408 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:44.177278 32632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d34158b1f040a196c9ce63943aa11f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 46227 } }
I20260807 03:01:44.177817 32632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:44.177948 32632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d34158b1f040a196c9ce63943aa11f, State: Initialized, Role: FOLLOWER
I20260807 03:01:44.178715 32632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d34158b1f040a196c9ce63943aa11f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 46227 } }
I20260807 03:01:44.179015 32632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:44.179179 32632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:44.179363 32632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:44.182325 32632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d34158b1f040a196c9ce63943aa11f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 46227 } }
I20260807 03:01:44.182873 32632 leader_election.cc:302] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97d34158b1f040a196c9ce63943aa11f; no voters:
I20260807 03:01:44.183667 32632 leader_election.cc:288] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:44.183787 32645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:44.184170 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:44.175860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51679 (local address 127.1.121.126:46227)
0807 03:01:44.176072 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:44.176080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:44.177509 (+ 1429us) server_negotiation.cc:407] Connection header received
0807 03:01:44.177687 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:44.177694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:44.177782 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:44.177907 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:44.178542 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:44.179721 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:44.181156 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:44.181913 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:44.182071 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:44.183196 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:44.183202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:44.183207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:44.183231 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:44.183302 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:44.183329 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:44.183492 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:44.183743 (+ 251us) server_negotiation.cc:299] Negotiation successful
0807 03:01:44.183810 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:01:44.184743 32645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [term 1 LEADER]: Becoming Leader. State: Replica: 97d34158b1f040a196c9ce63943aa11f, State: Running, Role: LEADER
I20260807 03:01:44.185413 32645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d34158b1f040a196c9ce63943aa11f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 46227 } }
I20260807 03:01:44.187907 32632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:44.188565 32646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97d34158b1f040a196c9ce63943aa11f. Latest consensus state: current_term: 1 leader_uuid: "97d34158b1f040a196c9ce63943aa11f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d34158b1f040a196c9ce63943aa11f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 46227 } } }
I20260807 03:01:44.188702 32647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97d34158b1f040a196c9ce63943aa11f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d34158b1f040a196c9ce63943aa11f" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 46227 } } }
I20260807 03:01:44.189114 32646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:44.189195 32647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:44.191272 32651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:44.197789 32651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:44.208791 32651 catalog_manager.cc:1347] Generated new cluster ID: c08cddc93c1a4da99965edece11becac
I20260807 03:01:44.208989 32651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:44.230917 32651 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:44.232434 32651 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:44.244138 32651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97d34158b1f040a196c9ce63943aa11f: Generated new TSK 0
I20260807 03:01:44.245193 32651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:44.259236 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 32541
W20260807 03:01:44.260416 32626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33707
W20260807 03:01:44.261320 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 32541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:44.261483 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 32541
W20260807 03:01:44.286056 32516 connection.cc:535] client connection to 127.1.121.126:46227 recv error: Network error: failed to read from TLS socket (remote: 127.1.121.126:46227): Connection reset by peer (error 104)
2026-08-07T03:01:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:44 dist-test-slave-3h5h krb5kdc[32672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:44 dist-test-slave-3h5h krb5kdc[32672](info): set up 2 sockets
Aug 07 03:01:44 dist-test-slave-3h5h krb5kdc[32672](info): commencing operation
krb5kdc: starting...
W20260807 03:01:45.623145 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:45 dist-test-slave-3h5h krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:01:45.983878 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34819
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:34819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:46.350086 32686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:46.350401 32686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:46.350580 32686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:46.357867 32686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:46.357985 32686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:46.358078 32686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:46.358207 32686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:46.358278 32686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:46.368117 32686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:34819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:34819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:46.369544 32686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:46.370903 32686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:46.383431 32693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:46.383486 32698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:46.383402 32692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:46.384585 32686 server_base.cc:1034] running on GCE node
I20260807 03:01:46.385437 32686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:46.387099 32686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:46.388356 32686 hybrid_clock.cc:648] HybridClock initialized: now 1786071706388316 us; error 50 us; skew 500 ppm
I20260807 03:01:46.388716 32686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:46 dist-test-slave-3h5h krb5kdc[32672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071706, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:46.391996 32686 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:01:46.393697 32686 webserver.cc:460] Webserver started at http://127.1.121.126:44753/ using document root <none> and password file <none>
I20260807 03:01:46.394402 32686 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:46.394555 32686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:46.394873 32686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:46.397986 32686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "6d646f7eac5a4cd8be76174185fdbfaf"
format_stamp: "Formatted at 2026-08-07 03:01:46 on dist-test-slave-3h5h"
server_key: "844eee67b8a195c827c37b643c40878a"
server_key_iv: "38879a9cf758733d8feca4c242369967"
server_key_version: "encryptionkey@0"
I20260807 03:01:46.398787 32686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "6d646f7eac5a4cd8be76174185fdbfaf"
format_stamp: "Formatted at 2026-08-07 03:01:46 on dist-test-slave-3h5h"
server_key: "844eee67b8a195c827c37b643c40878a"
server_key_iv: "38879a9cf758733d8feca4c242369967"
server_key_version: "encryptionkey@0"
I20260807 03:01:46.404248 32686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:01:46.407823 32706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:46.409253 32686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:01:46.409448 32686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "6d646f7eac5a4cd8be76174185fdbfaf"
format_stamp: "Formatted at 2026-08-07 03:01:46 on dist-test-slave-3h5h"
server_key: "844eee67b8a195c827c37b643c40878a"
server_key_iv: "38879a9cf758733d8feca4c242369967"
server_key_version: "encryptionkey@0"
I20260807 03:01:46.409657 32686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:46.415634 32686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:46.419741 32686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:46.420045 32686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:46.440160 32686 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:34819
I20260807 03:01:46.440217 307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:34819 every 8 connection(s)
I20260807 03:01:46.441633 32686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:46.445643 309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:46.451490 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 32686
I20260807 03:01:46.451704 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:46.452157 1509 external_mini_cluster.cc:1428] Setting key ae64c44d928bbfe20de9514e166aada0
I20260807 03:01:46.457168 309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf: Bootstrap starting.
I20260807 03:01:46.460283 313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:46.454126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38305 (local address 127.1.121.126:34819)
0807 03:01:46.455064 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:46.455080 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:46.455126 (+ 46us) server_negotiation.cc:407] Connection header received
0807 03:01:46.456778 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:46.456801 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:46.457196 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:46.457560 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:46.458699 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:46.458706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:46.458718 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:46.458740 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:46.458797 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:46.458816 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:46.458981 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:46.459177 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 03:01:46.459381 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":412,"threads_started":1}
I20260807 03:01:46.461081 309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:46.462168 309 log.cc:824] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:46.464207 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:46.465360 309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf: No bootstrap required, opened a new log
I20260807 03:01:46.471519 313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:46.468768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39133 (local address 127.1.121.126:34819)
0807 03:01:46.469015 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:46.469024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:46.469542 (+ 518us) server_negotiation.cc:407] Connection header received
0807 03:01:46.469751 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:46.469756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:46.469814 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:46.469929 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:46.470850 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:46.470855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:46.470863 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:46.470886 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:46.470934 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:46.470949 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:46.471076 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:46.471181 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 03:01:46.471248 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:01:46.473883 309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d646f7eac5a4cd8be76174185fdbfaf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34819 } }
I20260807 03:01:46.474267 309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:46.474378 309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d646f7eac5a4cd8be76174185fdbfaf, State: Initialized, Role: FOLLOWER
I20260807 03:01:46.474956 309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d646f7eac5a4cd8be76174185fdbfaf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34819 } }
I20260807 03:01:46.475225 309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:46.475384 309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:46.475586 309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:46.478341 309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d646f7eac5a4cd8be76174185fdbfaf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34819 } }
I20260807 03:01:46.478891 309 leader_election.cc:302] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d646f7eac5a4cd8be76174185fdbfaf; no voters:
I20260807 03:01:46.479498 309 leader_election.cc:288] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:46.479651 325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:46.480384 325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [term 1 LEADER]: Becoming Leader. State: Replica: 6d646f7eac5a4cd8be76174185fdbfaf, State: Running, Role: LEADER
I20260807 03:01:46.481022 325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d646f7eac5a4cd8be76174185fdbfaf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34819 } }
I20260807 03:01:46.482122 309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:46.484822 327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d646f7eac5a4cd8be76174185fdbfaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d646f7eac5a4cd8be76174185fdbfaf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34819 } } }
I20260807 03:01:46.485581 327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:46.485059 328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d646f7eac5a4cd8be76174185fdbfaf. Latest consensus state: current_term: 1 leader_uuid: "6d646f7eac5a4cd8be76174185fdbfaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d646f7eac5a4cd8be76174185fdbfaf" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 34819 } } }
I20260807 03:01:46.490864 328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:46.492779 335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:46.498980 335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:46.507711 335 catalog_manager.cc:1347] Generated new cluster ID: d823b07a5c124ac2a800f06981507b12
I20260807 03:01:46.507850 335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:46.530077 335 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:46.530550 335 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:46.538573 335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d646f7eac5a4cd8be76174185fdbfaf: Generated new TSK 0
I20260807 03:01:46.539402 335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:46.550735 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 32686
W20260807 03:01:46.551896 301 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38305
W20260807 03:01:46.552627 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 32686: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:46.552785 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 32686
2026-08-07T03:01:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:46 dist-test-slave-3h5h krb5kdc[354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:46 dist-test-slave-3h5h krb5kdc[354](info): set up 2 sockets
Aug 07 03:01:46 dist-test-slave-3h5h krb5kdc[354](info): commencing operation
krb5kdc: starting...
W20260807 03:01:47.920760 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.322s user 0.009s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:48 dist-test-slave-3h5h krb5kdc[354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:01:48.287882 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:40375
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:38427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:40375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:48.644474 369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:48.644765 369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:48.644918 369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:48.652112 369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:48.652272 369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:48.652390 369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:48.652505 369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:48.652602 369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:48.662684 369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:38427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:40375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:40375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:48.664124 369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:48.665545 369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:48.678099 377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:48.678136 376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:48.678795 380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:48.679090 369 server_base.cc:1034] running on GCE node
I20260807 03:01:48.679809 369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:48.681226 369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:48.682451 369 hybrid_clock.cc:648] HybridClock initialized: now 1786071708682399 us; error 46 us; skew 500 ppm
I20260807 03:01:48.682837 369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:48 dist-test-slave-3h5h krb5kdc[354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071708, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:48.686123 369 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:01:48.687534 369 webserver.cc:460] Webserver started at http://127.1.121.126:39171/ using document root <none> and password file <none>
I20260807 03:01:48.688158 369 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:48.688261 369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:48.688525 369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:48.691493 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "b2e1b38259834674b18349969cdbceb7"
format_stamp: "Formatted at 2026-08-07 03:01:48 on dist-test-slave-3h5h"
server_key: "b6dab5e89d116b67010ace796ab5392f"
server_key_iv: "c28cb38ce8846762586b05ac03fd9f43"
server_key_version: "encryptionkey@0"
I20260807 03:01:48.692200 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "b2e1b38259834674b18349969cdbceb7"
format_stamp: "Formatted at 2026-08-07 03:01:48 on dist-test-slave-3h5h"
server_key: "b6dab5e89d116b67010ace796ab5392f"
server_key_iv: "c28cb38ce8846762586b05ac03fd9f43"
server_key_version: "encryptionkey@0"
I20260807 03:01:48.697270 369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 03:01:48.700690 387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:48.702080 369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:01:48.702302 369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "b2e1b38259834674b18349969cdbceb7"
format_stamp: "Formatted at 2026-08-07 03:01:48 on dist-test-slave-3h5h"
server_key: "b6dab5e89d116b67010ace796ab5392f"
server_key_iv: "c28cb38ce8846762586b05ac03fd9f43"
server_key_version: "encryptionkey@0"
I20260807 03:01:48.702508 369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:48.713585 369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:48.717424 369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:48.717737 369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:48.737211 369 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:40375
I20260807 03:01:48.737242 454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:40375 every 8 connection(s)
I20260807 03:01:48.738659 369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:48.742849 455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:48.742825 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 369
I20260807 03:01:48.743014 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:48.743608 1509 external_mini_cluster.cc:1428] Setting key 9cf09fc2b73b414d2b20e453409f1305
I20260807 03:01:48.758358 455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7: Bootstrap starting.
Aug 07 03:01:48 dist-test-slave-3h5h krb5kdc[354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:01:48.761782 455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:48.762904 455 log.cc:824] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:48.766005 455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7: No bootstrap required, opened a new log
I20260807 03:01:48.767438 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:48.745525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36207 (local address 127.1.121.126:40375)
0807 03:01:48.746381 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:48.746392 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:48.746438 (+ 46us) server_negotiation.cc:407] Connection header received
0807 03:01:48.748142 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:48.748168 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:48.748578 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:48.748998 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:48.750866 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:48.752120 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:48.754053 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:48.754774 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:48.754916 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:48.759994 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:48.760003 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:48.760019 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:48.760069 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:48.763340 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:48.764038 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:48.764046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:48.764052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:48.764234 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:48.764648 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:48.764652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:48.764655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:48.765257 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:48.765576 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:48.766050 (+ 474us) server_negotiation.cc:299] Negotiation successful
0807 03:01:48.766468 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":399,"threads_started":1}
I20260807 03:01:48.771351 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:48.774545 455 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1b38259834674b18349969cdbceb7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40375 } }
I20260807 03:01:48.775092 455 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:48.775219 455 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e1b38259834674b18349969cdbceb7, State: Initialized, Role: FOLLOWER
I20260807 03:01:48.775961 455 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1b38259834674b18349969cdbceb7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40375 } }
I20260807 03:01:48.776415 455 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:48.776582 455 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:48.776790 455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:48.779351 455 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1b38259834674b18349969cdbceb7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40375 } }
I20260807 03:01:48.779904 455 leader_election.cc:302] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2e1b38259834674b18349969cdbceb7; no voters:
I20260807 03:01:48.780604 455 leader_election.cc:288] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:48.780736 471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:48.782199 471 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [term 1 LEADER]: Becoming Leader. State: Replica: b2e1b38259834674b18349969cdbceb7, State: Running, Role: LEADER
I20260807 03:01:48.782892 471 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1b38259834674b18349969cdbceb7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40375 } }
I20260807 03:01:48.783532 455 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:48.786008 473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2e1b38259834674b18349969cdbceb7. Latest consensus state: current_term: 1 leader_uuid: "b2e1b38259834674b18349969cdbceb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1b38259834674b18349969cdbceb7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40375 } } }
I20260807 03:01:48.786578 473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:48.794757 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:48.776008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47071 (local address 127.1.121.126:40375)
0807 03:01:48.776231 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:48.776240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:48.776676 (+ 436us) server_negotiation.cc:407] Connection header received
0807 03:01:48.776928 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:48.776936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:48.777015 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:48.777138 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:48.778678 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:48.779407 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:48.780746 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:48.781724 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:48.781856 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:48.784733 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:48.784743 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:48.784747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:48.784786 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:48.788983 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:48.790115 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:48.790124 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:48.790129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:48.790314 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:48.790909 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:48.790917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:48.790922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:48.791447 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:48.791841 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:48.793357 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0807 03:01:48.793657 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:01:48.791612 472 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2e1b38259834674b18349969cdbceb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1b38259834674b18349969cdbceb7" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40375 } } }
I20260807 03:01:48.795703 472 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:48.796411 479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:48.803400 479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:48.814157 479 catalog_manager.cc:1347] Generated new cluster ID: 4a41d90d1ccf4c9c9fabbf5b5c7b2788
I20260807 03:01:48.814325 479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:48.852378 479 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:48.853838 479 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:48.861442 479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2e1b38259834674b18349969cdbceb7: Generated new TSK 0
I20260807 03:01:48.862327 479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:48.870951 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 369
I20260807 03:01:48.873047 448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:48.970148 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 369
2026-08-07T03:01:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:49 dist-test-slave-3h5h krb5kdc[502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:49 dist-test-slave-3h5h krb5kdc[502](info): set up 2 sockets
Aug 07 03:01:49 dist-test-slave-3h5h krb5kdc[502](info): commencing operation
krb5kdc: starting...
W20260807 03:01:50.341642 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.007s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:50 dist-test-slave-3h5h krb5kdc[502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:01:50.699898 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:40213
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:40213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:51.075027 520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:51.075364 520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:51.075472 520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:51.083012 520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:51.083153 520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:51.083209 520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:51.083319 520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:51.083467 520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:51.094062 520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:40213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:40213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:51.095646 520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:51.097062 520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:51.109328 527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:51.109411 531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:51.109380 529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:51.110327 520 server_base.cc:1034] running on GCE node
I20260807 03:01:51.111124 520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:51.112478 520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:51.113689 520 hybrid_clock.cc:648] HybridClock initialized: now 1786071711113645 us; error 29 us; skew 500 ppm
I20260807 03:01:51.114135 520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:51 dist-test-slave-3h5h krb5kdc[502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071711, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:51.117323 520 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:01:51.118857 520 webserver.cc:460] Webserver started at http://127.1.121.126:39363/ using document root <none> and password file <none>
I20260807 03:01:51.119576 520 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:51.119747 520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:51.120064 520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:51.123224 520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "664793cbe7ea4ffaa48b332c1f93ecf8"
format_stamp: "Formatted at 2026-08-07 03:01:51 on dist-test-slave-3h5h"
server_key: "84586141ec762ced821bb9e573d6558a"
server_key_iv: "39a037a5fd629ae21720db60a043c71b"
server_key_version: "encryptionkey@0"
I20260807 03:01:51.124020 520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "664793cbe7ea4ffaa48b332c1f93ecf8"
format_stamp: "Formatted at 2026-08-07 03:01:51 on dist-test-slave-3h5h"
server_key: "84586141ec762ced821bb9e573d6558a"
server_key_iv: "39a037a5fd629ae21720db60a043c71b"
server_key_version: "encryptionkey@0"
I20260807 03:01:51.129282 520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260807 03:01:51.132757 540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:51.134107 520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:01:51.134321 520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "664793cbe7ea4ffaa48b332c1f93ecf8"
format_stamp: "Formatted at 2026-08-07 03:01:51 on dist-test-slave-3h5h"
server_key: "84586141ec762ced821bb9e573d6558a"
server_key_iv: "39a037a5fd629ae21720db60a043c71b"
server_key_version: "encryptionkey@0"
I20260807 03:01:51.134526 520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:51.147434 520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:51.151829 520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:51.152139 520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:51.171463 520 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:40213
I20260807 03:01:51.171471 609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:40213 every 8 connection(s)
I20260807 03:01:51.173023 520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:51.175318 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 520
I20260807 03:01:51.175518 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:51.175967 1509 external_mini_cluster.cc:1428] Setting key ae724b6bc65c06c7a83193cf59fc7fa0
I20260807 03:01:51.177454 610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:51.191375 615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:51.177987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45803 (local address 127.1.121.126:40213)
0807 03:01:51.178823 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:51.178838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:51.178890 (+ 52us) server_negotiation.cc:407] Connection header received
0807 03:01:51.181028 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:51.181059 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:51.181605 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:51.182054 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:51.182593 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:51.184665 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:51.186042 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:51.187081 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:51.187279 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:51.188437 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:51.188447 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:51.188466 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:51.189281 (+ 815us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:51.189381 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:51.189419 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:51.189552 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:51.189746 (+ 194us) server_negotiation.cc:299] Negotiation successful
0807 03:01:51.190038 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":353,"threads_started":1}
I20260807 03:01:51.192762 610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8: Bootstrap starting.
I20260807 03:01:51.195559 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:51.196660 610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:51.197973 610 log.cc:824] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:51.201064 610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8: No bootstrap required, opened a new log
I20260807 03:01:51.207929 615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:51.201055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40651 (local address 127.1.121.126:40213)
0807 03:01:51.201238 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:51.201245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:51.201779 (+ 534us) server_negotiation.cc:407] Connection header received
0807 03:01:51.201940 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:51.201946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:51.202025 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:51.202133 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:51.202661 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:51.203830 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:51.205253 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:51.205936 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:51.206082 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:51.207122 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:51.207129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:51.207135 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:51.207161 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:51.207213 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:51.207238 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:51.207338 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:51.207462 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 03:01:51.207565 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:01:51.210795 610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664793cbe7ea4ffaa48b332c1f93ecf8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40213 } }
I20260807 03:01:51.211189 610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:51.211318 610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664793cbe7ea4ffaa48b332c1f93ecf8, State: Initialized, Role: FOLLOWER
I20260807 03:01:51.211885 610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664793cbe7ea4ffaa48b332c1f93ecf8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40213 } }
I20260807 03:01:51.212127 610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:51.212288 610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:51.212468 610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:51.215260 610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664793cbe7ea4ffaa48b332c1f93ecf8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40213 } }
I20260807 03:01:51.215756 610 leader_election.cc:302] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 664793cbe7ea4ffaa48b332c1f93ecf8; no voters:
I20260807 03:01:51.216352 610 leader_election.cc:288] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:51.216500 625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:51.217341 625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [term 1 LEADER]: Becoming Leader. State: Replica: 664793cbe7ea4ffaa48b332c1f93ecf8, State: Running, Role: LEADER
I20260807 03:01:51.217895 625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664793cbe7ea4ffaa48b332c1f93ecf8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40213 } }
I20260807 03:01:51.218657 610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:51.220649 631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 664793cbe7ea4ffaa48b332c1f93ecf8. Latest consensus state: current_term: 1 leader_uuid: "664793cbe7ea4ffaa48b332c1f93ecf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664793cbe7ea4ffaa48b332c1f93ecf8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40213 } } }
I20260807 03:01:51.221071 631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:51.221112 630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "664793cbe7ea4ffaa48b332c1f93ecf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664793cbe7ea4ffaa48b332c1f93ecf8" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40213 } } }
I20260807 03:01:51.221611 630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:51.223376 634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:51.228557 634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:51.237504 634 catalog_manager.cc:1347] Generated new cluster ID: 2622050491c941cbbd2a821265fc7fb9
I20260807 03:01:51.237627 634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:51.274290 634 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:51.276252 634 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:51.293416 634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 664793cbe7ea4ffaa48b332c1f93ecf8: Generated new TSK 0
I20260807 03:01:51.294335 634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:51.302469 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 520
W20260807 03:01:51.304455 601 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45803
W20260807 03:01:51.305312 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:51.305466 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 520
2026-08-07T03:01:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:51 dist-test-slave-3h5h krb5kdc[656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:51 dist-test-slave-3h5h krb5kdc[656](info): set up 2 sockets
Aug 07 03:01:51 dist-test-slave-3h5h krb5kdc[656](info): commencing operation
krb5kdc: starting...
W20260807 03:01:52.657680 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.305s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:52 dist-test-slave-3h5h krb5kdc[656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:01:53.029953 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:40141
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:53.395489 674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:53.395784 674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:53.395974 674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:53.403549 674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:53.403702 674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:53.403806 674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:53.403913 674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:53.404011 674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:53.414119 674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:40141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:53.415508 674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:53.416905 674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:53.429347 681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:53.429384 682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:53.429575 684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:53.430215 674 server_base.cc:1034] running on GCE node
I20260807 03:01:53.431010 674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:53.432420 674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:53.433672 674 hybrid_clock.cc:648] HybridClock initialized: now 1786071713433620 us; error 65 us; skew 500 ppm
I20260807 03:01:53.434077 674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:53 dist-test-slave-3h5h krb5kdc[656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071713, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:53.437253 674 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:01:53.438716 674 webserver.cc:460] Webserver started at http://127.1.121.126:37831/ using document root <none> and password file <none>
I20260807 03:01:53.439390 674 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:53.439549 674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:53.439862 674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:53.442920 674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "975d0fc8677c46368faca6bb8b36ac42"
format_stamp: "Formatted at 2026-08-07 03:01:53 on dist-test-slave-3h5h"
server_key: "0f22362908e0a6f90eeb313b4547b1ff"
server_key_iv: "a71a2951e6642b4b4b83126a285dd682"
server_key_version: "encryptionkey@0"
I20260807 03:01:53.443719 674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "975d0fc8677c46368faca6bb8b36ac42"
format_stamp: "Formatted at 2026-08-07 03:01:53 on dist-test-slave-3h5h"
server_key: "0f22362908e0a6f90eeb313b4547b1ff"
server_key_iv: "a71a2951e6642b4b4b83126a285dd682"
server_key_version: "encryptionkey@0"
I20260807 03:01:53.449066 674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260807 03:01:53.452826 695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:53.454203 674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:01:53.454384 674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "975d0fc8677c46368faca6bb8b36ac42"
format_stamp: "Formatted at 2026-08-07 03:01:53 on dist-test-slave-3h5h"
server_key: "0f22362908e0a6f90eeb313b4547b1ff"
server_key_iv: "a71a2951e6642b4b4b83126a285dd682"
server_key_version: "encryptionkey@0"
I20260807 03:01:53.454595 674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:53.460453 674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:53.464517 674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:53.464978 674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:53.484440 674 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:40141
I20260807 03:01:53.484473 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:40141 every 8 connection(s)
I20260807 03:01:53.485960 674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:53.489917 770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:53.495878 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 674
I20260807 03:01:53.496076 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:53.496641 1509 external_mini_cluster.cc:1428] Setting key 25081c0322ca8cd324c11b116f6d9bd5
I20260807 03:01:53.501830 770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42: Bootstrap starting.
I20260807 03:01:53.505138 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:53.498741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52721 (local address 127.1.121.126:40141)
0807 03:01:53.499660 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:53.499672 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:53.499724 (+ 52us) server_negotiation.cc:407] Connection header received
0807 03:01:53.501533 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:53.501558 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:53.501963 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:53.502382 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:53.503485 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:53.503492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:53.503508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:53.503522 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:53.503585 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:53.503604 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:53.503844 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:53.503987 (+ 143us) server_negotiation.cc:299] Negotiation successful
0807 03:01:53.504179 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":420,"threads_started":1}
I20260807 03:01:53.505452 770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:53.506609 770 log.cc:824] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:53.508931 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:53.509708 770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42: No bootstrap required, opened a new log
I20260807 03:01:53.515986 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:53.513372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52831 (local address 127.1.121.126:40141)
0807 03:01:53.513594 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:53.513602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:53.513907 (+ 305us) server_negotiation.cc:407] Connection header received
0807 03:01:53.514108 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:53.514113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:53.514171 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:53.514285 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:53.515307 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:53.515314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:53.515323 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:01:53.515337 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:53.515377 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:01:53.515392 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:53.515559 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:53.515645 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 03:01:53.515706 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:01:53.518217 770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975d0fc8677c46368faca6bb8b36ac42" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40141 } }
I20260807 03:01:53.518620 770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:01:53.518733 770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975d0fc8677c46368faca6bb8b36ac42, State: Initialized, Role: FOLLOWER
I20260807 03:01:53.519299 770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975d0fc8677c46368faca6bb8b36ac42" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40141 } }
I20260807 03:01:53.519572 770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:01:53.519722 770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:01:53.519902 770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:53.522313 770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975d0fc8677c46368faca6bb8b36ac42" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40141 } }
I20260807 03:01:53.522884 770 leader_election.cc:302] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 975d0fc8677c46368faca6bb8b36ac42; no voters:
I20260807 03:01:53.523638 770 leader_election.cc:288] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:01:53.523742 782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:53.524544 782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [term 1 LEADER]: Becoming Leader. State: Replica: 975d0fc8677c46368faca6bb8b36ac42, State: Running, Role: LEADER
I20260807 03:01:53.525128 782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975d0fc8677c46368faca6bb8b36ac42" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40141 } }
I20260807 03:01:53.526011 770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:53.527921 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "975d0fc8677c46368faca6bb8b36ac42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975d0fc8677c46368faca6bb8b36ac42" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40141 } } }
I20260807 03:01:53.528028 786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 975d0fc8677c46368faca6bb8b36ac42. Latest consensus state: current_term: 1 leader_uuid: "975d0fc8677c46368faca6bb8b36ac42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975d0fc8677c46368faca6bb8b36ac42" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 40141 } } }
I20260807 03:01:53.528332 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:53.528412 786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42 [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:53.529999 791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:01:53.535549 791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:53.544876 791 catalog_manager.cc:1347] Generated new cluster ID: 446acf5df6dc4ffb9cae1c48b369fd06
I20260807 03:01:53.544998 791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:53.583127 791 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:53.583600 791 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:53.590216 791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 975d0fc8677c46368faca6bb8b36ac42: Generated new TSK 0
I20260807 03:01:53.591012 791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:53.598917 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 674
W20260807 03:01:53.600137 762 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52721
W20260807 03:01:53.600860 1509 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:01:53.601025 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 674
2026-08-07T03:01:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2296 ms)
[----------] 18 tests from AuthTokenIssuingTest (42493 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:53 dist-test-slave-3h5h krb5kdc[814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:53 dist-test-slave-3h5h krb5kdc[814](info): set up 2 sockets
Aug 07 03:01:53 dist-test-slave-3h5h krb5kdc[814](info): commencing operation
krb5kdc: starting...
W20260807 03:01:54.939992 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:01:55 dist-test-slave-3h5h krb5kdc[814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:01:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:01:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:01:55.321064 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:36041
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:36041,127.1.121.125:45405,127.1.121.124:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:01:55.680307 830 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:01:55.680699 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:55.680848 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:55.681025 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:55.688208 830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:55.688290 830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:55.688414 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:55.688503 830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:55.688596 830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:55.698457 830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:36041,127.1.121.125:45405,127.1.121.124:39367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:55.699801 830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:55.701174 830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:55.713493 837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:55.713411 838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:55.713457 841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:55.714437 830 server_base.cc:1034] running on GCE node
I20260807 03:01:55.715229 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:55.716598 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:55.717864 830 hybrid_clock.cc:648] HybridClock initialized: now 1786071715717787 us; error 42 us; skew 500 ppm
I20260807 03:01:55.718217 830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:55 dist-test-slave-3h5h krb5kdc[814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071715, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:55.725142 830 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:01:55.727036 830 webserver.cc:460] Webserver started at http://127.1.121.126:34175/ using document root <none> and password file <none>
I20260807 03:01:55.727790 830 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:55.727932 830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:55.728288 830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:55.731457 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "3bc6671e0b944fd1b64ae65332362bac"
format_stamp: "Formatted at 2026-08-07 03:01:55 on dist-test-slave-3h5h"
server_key: "5f28945cbb89859f8d16630cc6f86bd9"
server_key_iv: "ba7bbceb7a005fec8f601c87728a0a34"
server_key_version: "encryptionkey@0"
I20260807 03:01:55.732261 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "3bc6671e0b944fd1b64ae65332362bac"
format_stamp: "Formatted at 2026-08-07 03:01:55 on dist-test-slave-3h5h"
server_key: "5f28945cbb89859f8d16630cc6f86bd9"
server_key_iv: "ba7bbceb7a005fec8f601c87728a0a34"
server_key_version: "encryptionkey@0"
I20260807 03:01:55.737614 830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 03:01:55.741245 851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:55.742586 830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:01:55.742785 830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "3bc6671e0b944fd1b64ae65332362bac"
format_stamp: "Formatted at 2026-08-07 03:01:55 on dist-test-slave-3h5h"
server_key: "5f28945cbb89859f8d16630cc6f86bd9"
server_key_iv: "ba7bbceb7a005fec8f601c87728a0a34"
server_key_version: "encryptionkey@0"
I20260807 03:01:55.743031 830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:55.774791 830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:55.779026 830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:55.779340 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:55.800480 923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:36041 every 8 connection(s)
I20260807 03:01:55.800518 830 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:36041
I20260807 03:01:55.802104 830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:01:55.807327 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:55.807331 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 830
I20260807 03:01:55.807574 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:01:55.808288 1509 external_mini_cluster.cc:1428] Setting key 7502be7691a3afb5a73c4926ecd241f3
I20260807 03:01:55.818397 924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:01:55 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071715, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.121.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.125@KRBTEST.COM" created.
I20260807 03:01:55.855894 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:55.824331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.1.121.126:36041)
0807 03:01:55.825069 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:55.825080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:55.825142 (+ 62us) server_negotiation.cc:407] Connection header received
0807 03:01:55.827693 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:55.827705 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:55.828069 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:55.828322 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:55.830215 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:55.832320 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:55.834795 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:55.835832 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:55.836025 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:55.845943 (+ 9918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:55.845956 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:55.845980 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:55.846028 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:55.849944 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:55.851091 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:55.851100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:55.851105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:55.851333 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:55.851906 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:55.851913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:55.851918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:55.852814 (+ 896us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:55.853082 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:55.853882 (+ 800us) server_negotiation.cc:299] Negotiation successful
0807 03:01:55.854278 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":347,"threads_started":1}
I20260807 03:01:55.855916 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:55.823055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:36041 (local address 127.0.0.1:33536)
0807 03:01:55.824099 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:55.824156 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:55.827195 (+ 3039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:55.828393 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:55.828403 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:55.829322 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:55.829915 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:55.829945 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:55.832525 (+ 2580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:55.832534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:55.834609 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:55.834629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:55.836402 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:55.836412 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:55.836651 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:55.841664 (+ 5013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:55.841705 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:55.845585 (+ 3880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:55.850105 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:55.850123 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:55.850140 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:55.850878 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:55.851488 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:55.851495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:55.851500 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:55.851755 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:55.853015 (+ 1260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:55.853027 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:55.853348 (+ 321us) client_negotiation.cc:769] Sending connection context
0807 03:01:55.853665 (+ 317us) client_negotiation.cc:240] Negotiation successful
0807 03:01:55.854543 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":204,"thread_start_us":405,"threads_started":1}
I20260807 03:01:55.859437 924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:01:55.860728 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:55.860193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34140)
0807 03:01:55.860406 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:55.860548 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 03:01:55.861052 855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
I20260807 03:01:55.862829 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:55.862378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34148)
0807 03:01:55.862560 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:55.862656 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 03:01:55.862998 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:45405: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab.
Entry for principal kudu/127.1.121.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab.
I20260807 03:01:55.896306 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.125:45405
--webserver_interface=127.1.121.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab
--principal=kudu/127.1.121.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:36041,127.1.121.125:45405,127.1.121.124:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:01:55.912395 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } attempt: 1
I20260807 03:01:55.913899 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:55.913260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34164)
0807 03:01:55.913625 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:55.913763 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260807 03:01:55.915015 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:55.914613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34166)
0807 03:01:55.914763 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:55.914873 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 03:01:55.915280 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:45405: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
I20260807 03:01:55.983634 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } attempt: 2
I20260807 03:01:55.984876 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:55.984371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34180)
0807 03:01:55.984574 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:55.984686 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 03:01:55.985898 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:55.985549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34184)
0807 03:01:55.985697 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:55.985783 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 03:01:55.986083 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:45405: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
I20260807 03:01:56.077379 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } attempt: 3
I20260807 03:01:56.078583 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.078115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34186)
0807 03:01:56.078347 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.078449 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 03:01:56.079816 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.079442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34188)
0807 03:01:56.079623 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.079694 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 03:01:56.080062 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:45405: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
I20260807 03:01:56.223384 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } attempt: 4
I20260807 03:01:56.225015 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.224281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34204)
0807 03:01:56.224687 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.224793 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260807 03:01:56.226281 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.225849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34216)
0807 03:01:56.226043 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.226141 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 03:01:56.226574 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:45405: Network error: Client connection negotiation failed: client connection to 127.1.121.125:45405: connect: Connection refused (error 111)
W20260807 03:01:56.260962 931 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:01:56.261338 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:56.261464 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:56.261603 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:56.269107 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:56.269203 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:56.269291 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:56.269394 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:56.269508 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:56.279711 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:36041,127.1.121.125:45405,127.1.121.124:39367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab
--principal=kudu/127.1.121.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.125:45405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.121.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:56.281090 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:56.282543 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:56.294755 939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:56.294723 940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:56.294806 943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:56.295850 931 server_base.cc:1034] running on GCE node
I20260807 03:01:56.296706 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:56.298190 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:56.299418 931 hybrid_clock.cc:648] HybridClock initialized: now 1786071716299330 us; error 56 us; skew 500 ppm
I20260807 03:01:56.299772 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071716, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:56.303061 931 init.cc:377] Logged in from keytab as kudu/127.1.121.125@KRBTEST.COM (short username kudu)
I20260807 03:01:56.304514 931 webserver.cc:460] Webserver started at http://127.1.121.125:37131/ using document root <none> and password file <none>
I20260807 03:01:56.305298 931 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:56.305441 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:56.305776 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:56.308969 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/data/instance:
uuid: "50cf39412cdc4a91b2360d20d371dcb1"
format_stamp: "Formatted at 2026-08-07 03:01:56 on dist-test-slave-3h5h"
server_key: "f23a553365780cf931de81106b0ecef7"
server_key_iv: "eb6a426ec1e017ae10b51dcde63cb212"
server_key_version: "encryptionkey@0"
I20260807 03:01:56.309808 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/wal/instance:
uuid: "50cf39412cdc4a91b2360d20d371dcb1"
format_stamp: "Formatted at 2026-08-07 03:01:56 on dist-test-slave-3h5h"
server_key: "f23a553365780cf931de81106b0ecef7"
server_key_iv: "eb6a426ec1e017ae10b51dcde63cb212"
server_key_version: "encryptionkey@0"
I20260807 03:01:56.315080 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 03:01:56.318560 954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:56.319926 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:01:56.320122 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/wal
uuid: "50cf39412cdc4a91b2360d20d371dcb1"
format_stamp: "Formatted at 2026-08-07 03:01:56 on dist-test-slave-3h5h"
server_key: "f23a553365780cf931de81106b0ecef7"
server_key_iv: "eb6a426ec1e017ae10b51dcde63cb212"
server_key_version: "encryptionkey@0"
I20260807 03:01:56.320353 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:56.331826 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:56.336378 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:56.336666 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:56.356107 931 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.125:45405
I20260807 03:01:56.356127 1023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.125:45405 every 8 connection(s)
I20260807 03:01:56.357652 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
I20260807 03:01:56.361397 1024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:56.365242 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 931
I20260807 03:01:56.365444 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-1/wal/instance
I20260807 03:01:56.366027 1509 external_mini_cluster.cc:1428] Setting key d8107f194f5226d31bf4ab3a4124e4dd
I20260807 03:01:56.369897 1024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071716, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:01:56.394393 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.373378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.1.121.126:36041)
0807 03:01:56.373932 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.373938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.375019 (+ 1081us) server_negotiation.cc:407] Connection header received
0807 03:01:56.377964 (+ 2945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.377968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.378038 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.378138 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.380398 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.381192 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.383192 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.383923 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.384075 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.388014 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:56.388020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:56.388023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:56.388071 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:56.390885 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.391947 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.391950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.391953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.392072 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.392555 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.392558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.392560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.392818 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:56.392995 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:56.393933 (+ 938us) server_negotiation.cc:299] Negotiation successful
0807 03:01:56.394102 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":281,"threads_started":1}
I20260807 03:01:56.395913 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.373973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:36041 (local address 127.0.0.1:33538)
0807 03:01:56.374833 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.374890 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:56.377646 (+ 2756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:56.378408 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:56.378420 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:56.379258 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:56.380160 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.380189 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.381359 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.381367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.383013 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.383027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.384134 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.384139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.384370 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.385117 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:56.385139 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:56.387761 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:56.391036 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.391049 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.391068 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.391821 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.392192 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.392199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.392205 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.392443 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.393014 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:56.393026 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:56.393512 (+ 486us) client_negotiation.cc:769] Sending connection context
0807 03:01:56.393851 (+ 339us) client_negotiation.cc:240] Negotiation successful
0807 03:01:56.394425 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":313,"threads_started":1}
I20260807 03:01:56.397799 1024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.121.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.124@KRBTEST.COM" created.
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071716, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 03:01:56.414460 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.398766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.1.121.125:45405)
0807 03:01:56.399562 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.399573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.399607 (+ 34us) server_negotiation.cc:407] Connection header received
0807 03:01:56.399668 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.399676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.399938 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.400105 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.400800 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.402646 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.403817 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.404555 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.404657 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.408502 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:56.408513 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:56.408527 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:56.408556 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:56.411421 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.412266 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.412272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.412274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.412385 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.412811 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.412814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.412816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.413219 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:56.413381 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:56.413840 (+ 459us) server_negotiation.cc:299] Negotiation successful
0807 03:01:56.414155 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":586,"thread_start_us":434,"threads_started":1}
I20260807 03:01:56.414544 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.398489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34224)
0807 03:01:56.398702 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.398722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:56.398835 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:56.400106 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:56.400110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:56.400468 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:56.400611 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.400620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.402791 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.402795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.403654 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.403665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.404700 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.404705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.404891 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.405466 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:56.405483 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:56.408283 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:56.411567 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.411572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.411576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.412125 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.412495 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.412501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.412508 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.412700 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.413351 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:56.413358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:56.413526 (+ 168us) client_negotiation.cc:769] Sending connection context
0807 03:01:56.413702 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 03:01:56.414034 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 03:01:56.416911 1024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:01:56.418291 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.417700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33662)
0807 03:01:56.417923 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.418072 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260807 03:01:56.418593 960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
I20260807 03:01:56.420287 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.419847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33678)
0807 03:01:56.420016 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.420121 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:01:56.420509 1024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:39367: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab.
Entry for principal kudu/127.1.121.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab.
I20260807 03:01:56.448769 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.124:39367
--webserver_interface=127.1.121.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab
--principal=kudu/127.1.121.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:36041,127.1.121.125:45405,127.1.121.124:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:01:56.469851 1024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } attempt: 1
I20260807 03:01:56.471309 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.470806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33684)
0807 03:01:56.471027 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.471137 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 03:01:56.472427 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.472101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33700)
0807 03:01:56.472251 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.472325 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 03:01:56.472723 1024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:39367: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
I20260807 03:01:56.525880 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } attempt: 5
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071715, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 03:01:56.540972 1024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } attempt: 2
I20260807 03:01:56.542160 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.541641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33702)
0807 03:01:56.541853 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.541964 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 03:01:56.542804 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.526794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.1.121.125:45405)
0807 03:01:56.526999 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.527003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.527098 (+ 95us) server_negotiation.cc:407] Connection header received
0807 03:01:56.527285 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.527289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.527371 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.527460 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.528609 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.529560 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.531366 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.532103 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.532217 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.536323 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:56.536328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:56.536332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:56.536367 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:56.539388 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.540283 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.540289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.540292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.540450 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.541032 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.541038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.541041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.541456 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:56.541629 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:56.542148 (+ 519us) server_negotiation.cc:299] Negotiation successful
0807 03:01:56.542440 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 03:01:56.542948 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.526726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34226)
0807 03:01:56.526986 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.527010 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:56.527162 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:56.527541 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:56.527546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:56.528111 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:56.528393 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.528408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.529725 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.529732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.531189 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.531207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.532275 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.532281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.532528 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.533343 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:56.533367 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:56.536116 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:56.539538 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.539543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.539547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.540110 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.540648 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.540656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.540662 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.540883 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.541591 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:56.541596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:56.541792 (+ 196us) client_negotiation.cc:769] Sending connection context
0807 03:01:56.541966 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 03:01:56.542297 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260807 03:01:56.543430 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.542981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33708)
0807 03:01:56.543208 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.543292 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260807 03:01:56.543711 1024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:39367: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
I20260807 03:01:56.544561 924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:01:56.545694 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.545304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33712)
0807 03:01:56.545470 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.545557 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 03:01:56.546787 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.546437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33716)
0807 03:01:56.546586 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.546670 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 03:01:56.547057 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:39367: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
I20260807 03:01:56.598295 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } attempt: 1
I20260807 03:01:56.599535 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.599055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33720)
0807 03:01:56.599266 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.599363 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 03:01:56.600585 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.600254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33732)
0807 03:01:56.600406 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.600486 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 03:01:56.600801 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:39367: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
I20260807 03:01:56.634966 1024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } attempt: 3
I20260807 03:01:56.636225 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.635728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33742)
0807 03:01:56.635952 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.636047 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 03:01:56.637425 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.636989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33744)
0807 03:01:56.637226 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.637305 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260807 03:01:56.637647 1024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:39367: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
I20260807 03:01:56.669166 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } attempt: 2
I20260807 03:01:56.670430 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.669925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33754)
0807 03:01:56.670169 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.670260 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 03:01:56.671504 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.671153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33764)
0807 03:01:56.671314 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.671391 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 03:01:56.671769 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:39367: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
I20260807 03:01:56.778012 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } attempt: 3
I20260807 03:01:56.779294 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.778815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33780)
0807 03:01:56.779025 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.779119 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 03:01:56.780464 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.780075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33784)
0807 03:01:56.780289 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.780370 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260807 03:01:56.780754 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:39367: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
I20260807 03:01:56.780881 1024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } attempt: 4
I20260807 03:01:56.782044 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.781456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33788)
0807 03:01:56.781731 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.781840 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 03:01:56.783212 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.782844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33796)
0807 03:01:56.783000 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.783084 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 03:01:56.783434 1024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:39367: Network error: Client connection negotiation failed: client connection to 127.1.121.124:39367: connect: Connection refused (error 111)
W20260807 03:01:56.811170 1037 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:01:56.811509 1037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:01:56.811656 1037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:01:56.811775 1037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:01:56.819145 1037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:01:56.819232 1037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:01:56.819350 1037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:01:56.819427 1037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:01:56.819538 1037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:01:56.829823 1037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:36041,127.1.121.125:45405,127.1.121.124:39367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab
--principal=kudu/127.1.121.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.124:39367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.121.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:01:56.831204 1037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:01:56.832626 1037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:01:56.845541 1043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:56.845588 1044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:01:56.845428 1047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:01:56.850426 1037 server_base.cc:1034] running on GCE node
I20260807 03:01:56.851281 1037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:01:56.852783 1037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:01:56.854068 1037 hybrid_clock.cc:648] HybridClock initialized: now 1786071716854006 us; error 39 us; skew 500 ppm
I20260807 03:01:56.854423 1037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071716, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:01:56.857745 1037 init.cc:377] Logged in from keytab as kudu/127.1.121.124@KRBTEST.COM (short username kudu)
I20260807 03:01:56.859360 1037 webserver.cc:460] Webserver started at http://127.1.121.124:37135/ using document root <none> and password file <none>
I20260807 03:01:56.860111 1037 fs_manager.cc:359] Metadata directory not provided
I20260807 03:01:56.860255 1037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:01:56.860595 1037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:01:56.863709 1037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/data/instance:
uuid: "b120e2e3535744c1bc0e4a34e6edaefd"
format_stamp: "Formatted at 2026-08-07 03:01:56 on dist-test-slave-3h5h"
server_key: "5655674803679ef9fdf369bce0882d9c"
server_key_iv: "bc11c9cddde9d24b41ce4f9ee69cb8ef"
server_key_version: "encryptionkey@0"
I20260807 03:01:56.864568 1037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/wal/instance:
uuid: "b120e2e3535744c1bc0e4a34e6edaefd"
format_stamp: "Formatted at 2026-08-07 03:01:56 on dist-test-slave-3h5h"
server_key: "5655674803679ef9fdf369bce0882d9c"
server_key_iv: "bc11c9cddde9d24b41ce4f9ee69cb8ef"
server_key_version: "encryptionkey@0"
I20260807 03:01:56.870131 1037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 03:01:56.873759 1056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:01:56.875175 1037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:01:56.875378 1037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/wal
uuid: "b120e2e3535744c1bc0e4a34e6edaefd"
format_stamp: "Formatted at 2026-08-07 03:01:56 on dist-test-slave-3h5h"
server_key: "5655674803679ef9fdf369bce0882d9c"
server_key_iv: "bc11c9cddde9d24b41ce4f9ee69cb8ef"
server_key_version: "encryptionkey@0"
I20260807 03:01:56.875608 1037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:01:56.906313 1037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:01:56.910490 1037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:01:56.910782 1037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:01:56.930104 1037 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.124:39367
I20260807 03:01:56.930130 1120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.124:39367 every 8 connection(s)
I20260807 03:01:56.931559 1037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
I20260807 03:01:56.935456 1121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:01:56.935597 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 1037
I20260807 03:01:56.935809 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786071323612974-1509-0/minicluster-data/master-2/wal/instance
I20260807 03:01:56.936295 1509 external_mini_cluster.cc:1428] Setting key 7c7f4d62294db4d3d7d94396caa207b6
I20260807 03:01:56.945577 1121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071716, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:01:56.958034 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } attempt: 4
I20260807 03:01:56.961675 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.938205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.1.121.124:39367)
0807 03:01:56.939127 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.939142 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.939185 (+ 43us) server_negotiation.cc:407] Connection header received
0807 03:01:56.940699 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.940721 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.941169 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.941521 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.943413 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.944686 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.946570 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.947439 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.947638 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.952443 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:56.952450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:56.952461 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:56.952500 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:56.956021 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.956813 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.956876 (+ 63us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.956886 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.957075 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.957475 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.957482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.957485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.958192 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:56.958484 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:56.959578 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0807 03:01:56.960054 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":496,"threads_started":1}
I20260807 03:01:56.964972 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.948713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:36041 (local address 127.0.0.1:33544)
0807 03:01:56.949633 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.949692 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:56.949847 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:56.950274 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:56.950279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:56.950858 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:56.951105 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.951113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.952201 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.952209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.953404 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.953417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.954507 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.954519 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.954654 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.955281 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:56.955301 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:56.957682 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:56.961890 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.961899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.961906 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.962502 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.962988 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.962993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.962996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.963227 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.963768 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:56.963775 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:56.963967 (+ 192us) client_negotiation.cc:769] Sending connection context
0807 03:01:56.964144 (+ 177us) client_negotiation.cc:240] Negotiation successful
0807 03:01:56.964515 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":540,"threads_started":1}
I20260807 03:01:56.965708 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.948642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.1.121.126:36041)
0807 03:01:56.949256 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.949264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.949798 (+ 534us) server_negotiation.cc:407] Connection header received
0807 03:01:56.949986 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.949993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.950092 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.950234 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.951264 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.952036 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.953589 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.954309 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.954449 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.957881 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:56.957888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:56.957892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:56.957941 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:56.961721 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.962665 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.962671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.962675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.962859 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.963383 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.963386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.963388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.963629 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:56.963815 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:56.964589 (+ 774us) server_negotiation.cc:299] Negotiation successful
0807 03:01:56.964817 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":289,"threads_started":1}
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071715, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
I20260807 03:01:56.968261 1121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:01:56.975648 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.958670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33820)
0807 03:01:56.958833 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.958852 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:56.958991 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:56.959717 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:56.959722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:56.960216 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:56.960382 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.960391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.961809 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.961815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.962900 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.962915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.963846 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.963853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.964044 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.964645 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:56.964666 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:56.967614 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:56.972501 (+ 4887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.972506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.972510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.973097 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.973584 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.973592 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.973596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.973806 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.974502 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:56.974508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:56.974709 (+ 201us) client_negotiation.cc:769] Sending connection context
0807 03:01:56.974851 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 03:01:56.975171 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:01:56.977217 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.958728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.1.121.124:39367)
0807 03:01:56.959373 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.959379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.959400 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:01:56.959447 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.959451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.959521 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.960508 (+ 987us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.960719 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.961664 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.963066 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.963649 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.964359 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.967859 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:56.967866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:56.967870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:56.967905 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:56.972317 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.973256 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.973260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.973264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.973426 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.973972 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.973977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.973979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.974331 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:56.976343 (+ 2012us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:56.976507 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 03:01:56.976758 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":309,"threads_started":1}
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071716, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 03:01:56.985924 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.969076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:45405 (local address 127.0.0.1:34240)
0807 03:01:56.969222 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.969241 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:56.969364 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:56.969897 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:56.969902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:56.970625 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:56.970836 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.970852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.972234 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.972239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.973537 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.973552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.974907 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.974913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.975080 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.975637 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:56.975658 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:56.978474 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:56.983045 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.983050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.983054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.983568 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.984047 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.984054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.984060 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.984245 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:56.984967 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:56.984973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:56.985140 (+ 167us) client_negotiation.cc:769] Sending connection context
0807 03:01:56.985316 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 03:01:56.985592 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 03:01:56.986876 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.969220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.1.121.125:45405)
0807 03:01:56.969440 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.969448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.969468 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:01:56.969528 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.969534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.969611 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.969705 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.971033 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.972047 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.973993 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.974604 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.974719 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.978705 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:56.978711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:56.978733 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:56.978782 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:56.982868 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.983721 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.983729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.983732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.983903 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.984384 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.984389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.984392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.984737 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:56.984897 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:56.986306 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0807 03:01:56.986523 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:01:56.987839 1121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:01:56.987967 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac: Bootstrap starting.
I20260807 03:01:56.992580 924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:56.994185 924 log.cc:824] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:56.994709 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.966238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.1.121.126:36041)
0807 03:01:56.966943 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.966949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.966981 (+ 32us) server_negotiation.cc:407] Connection header received
0807 03:01:56.967021 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.967024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.967077 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.967213 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.969061 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.970293 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.972209 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.972928 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.973681 (+ 753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.979044 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:56.979050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:56.979056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:56.979100 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:56.990486 (+ 11386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.991444 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.991450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.991454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.991630 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.992128 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.992133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.992136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.992535 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:56.992809 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:56.993848 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0807 03:01:56.994101 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":329,"threads_started":1}
Aug 07 03:01:56 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071716, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
I20260807 03:01:57.000085 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac: No bootstrap required, opened a new log
I20260807 03:01:57.002065 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.988759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33836)
0807 03:01:56.989009 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:56.989022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:56.989146 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:56.989479 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:56.989483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:56.989864 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:56.989998 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.990007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.990958 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.990965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.992012 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:56.992024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.993040 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.993045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:56.993252 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.993773 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:56.993789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:56.996149 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:56.999335 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:56.999341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:56.999345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:56.999720 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:57.000133 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:57.000141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:57.000146 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:57.000325 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:57.000952 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:57.000959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:57.001131 (+ 172us) client_negotiation.cc:769] Sending connection context
0807 03:01:57.001301 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 03:01:57.001627 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260807 03:01:57.002245 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.988677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.1.121.124:39367)
0807 03:01:56.988879 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.988884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.989091 (+ 207us) server_negotiation.cc:407] Connection header received
0807 03:01:56.989280 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.989285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.989363 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.989463 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.990134 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.990832 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.992193 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:56.992875 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:56.993025 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:56.996347 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:56.996353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:56.996357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:56.996392 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:56.999185 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:56.999837 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:56.999844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:56.999848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:56.999991 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.000457 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.000462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.000466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.000799 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:57.000974 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:57.001425 (+ 451us) server_negotiation.cc:299] Negotiation successful
0807 03:01:57.001834 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":62}
Aug 07 03:01:57 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 03:01:57.008164 1121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd: Bootstrap starting.
I20260807 03:01:57.011924 924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } }
I20260807 03:01:57.012436 924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc6671e0b944fd1b64ae65332362bac, State: Initialized, Role: FOLLOWER
I20260807 03:01:57.012715 1121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:57.013088 924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } }
I20260807 03:01:57.014015 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:56.997354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.1.121.125:45405)
0807 03:01:56.997690 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:56.997696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:56.997715 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:01:56.997827 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:56.997832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:56.997909 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:56.998156 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:56.999978 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.001044 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.002810 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.003242 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.003388 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:57.007874 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:57.007879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:57.007883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:57.007924 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:57.010918 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.011707 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.011714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.011718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.011865 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.012293 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.012299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.012302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.012710 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:57.013022 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:57.013380 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 03:01:57.013631 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260807 03:01:57.014421 1121 log.cc:824] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:57.014955 1136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.015514 1136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:01:57.016639 924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:57.019071 1121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd: No bootstrap required, opened a new log
I20260807 03:01:57.020005 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:01:57.023733 886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3bc6671e0b944fd1b64ae65332362bac"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33554
I20260807 03:01:57.024338 886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:01:57.024570 886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:57.028475 886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } }
I20260807 03:01:57.030283 1121 raft_consensus.cc:348] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } }
I20260807 03:01:57.030756 1121 raft_consensus.cc:729] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b120e2e3535744c1bc0e4a34e6edaefd, State: Initialized, Role: FOLLOWER
I20260807 03:01:57.031206 886 leader_election.cc:288] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [CANDIDATE]: Term 1 election: Requested vote from peers 50cf39412cdc4a91b2360d20d371dcb1 (127.1.121.125:45405), b120e2e3535744c1bc0e4a34e6edaefd (127.1.121.124:39367)
I20260807 03:01:57.031317 1121 consensus_queue.cc:260] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } }
W20260807 03:01:57.032471 859 tablet.cc:2369] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:01:57.050771 1093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bc6671e0b944fd1b64ae65332362bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b120e2e3535744c1bc0e4a34e6edaefd"
I20260807 03:01:57.051304 1093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:57.037902 830 master_runner.cc:186] Error getting master registration for 127.1.121.125:45405: OK, instance_id { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" instance_seqno: 1786071716350673 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:01:57.052451 830 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 03:01:57.054132 1153 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:01:57.054275 1153 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:01:57.054736 1121 sys_catalog.cc:564] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:57.054977 1093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bc6671e0b944fd1b64ae65332362bac in term 1.
I20260807 03:01:57.055791 857 leader_election.cc:302] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc6671e0b944fd1b64ae65332362bac, b120e2e3535744c1bc0e4a34e6edaefd; no voters:
I20260807 03:01:57.056285 1136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:01:57.056895 1154 sys_catalog.cc:455] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.058060 1154 sys_catalog.cc:458] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:01:57.069624 1136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [term 1 LEADER]: Becoming Leader. State: Replica: 3bc6671e0b944fd1b64ae65332362bac, State: Running, Role: LEADER
I20260807 03:01:57.070458 1136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } }
I20260807 03:01:57.078337 1164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc6671e0b944fd1b64ae65332362bac. Latest consensus state: current_term: 1 leader_uuid: "3bc6671e0b944fd1b64ae65332362bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.079028 1164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:57.082628 1024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } attempt: 5
I20260807 03:01:57.092375 1182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 07 03:01:57 dist-test-slave-3h5h krb5kdc[814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071716, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
I20260807 03:01:57.094978 1182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:01:57.104261 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:57.083380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:39367 (local address 127.0.0.1:33846)
0807 03:01:57.083861 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0807 03:01:57.083887 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 03:01:57.084040 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:01:57.084952 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:01:57.084958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:01:57.085526 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:01:57.085783 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:57.085797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.087433 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.087439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:57.088902 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:01:57.088921 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.090216 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.090223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:01:57.090519 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:57.091271 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:01:57.091296 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:01:57.094560 (+ 3264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:01:57.099057 (+ 4497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:57.099064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:57.099070 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:57.099632 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:57.102081 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:01:57.102086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:01:57.102090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:01:57.102281 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:01:57.103142 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:01:57.103147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:01:57.103327 (+ 180us) client_negotiation.cc:769] Sending connection context
0807 03:01:57.103516 (+ 189us) client_negotiation.cc:240] Negotiation successful
0807 03:01:57.103857 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315}
I20260807 03:01:57.106206 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:57.084303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.1.121.124:39367)
0807 03:01:57.084499 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.084505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.084528 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:01:57.084631 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.084637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.084739 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.085958 (+ 1219us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.086138 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.087256 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.089088 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.089802 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.089949 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:57.094792 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:57.094800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:57.094806 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:57.094850 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:57.098857 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.100571 (+ 1714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.100577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.100581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.100780 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.102437 (+ 1657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.102443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.102447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.102887 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:57.103058 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:57.105564 (+ 2506us) server_negotiation.cc:299] Negotiation successful
0807 03:01:57.105812 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:01:57.115182 1024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1: Bootstrap starting.
I20260807 03:01:57.119690 1024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1: Neither blocks nor log segments found. Creating new log.
I20260807 03:01:57.126011 1024 log.cc:824] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1: Log is configured to *not* fsync() on all Append() calls
I20260807 03:01:57.131567 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:57.081251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.1.121.126:36041)
0807 03:01:57.081531 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.081540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.089794 (+ 8254us) server_negotiation.cc:407] Connection header received
0807 03:01:57.091573 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.091580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.091699 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.091775 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.105461 (+ 13686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.106998 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.109660 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.110736 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.110894 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:57.117046 (+ 6152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:57.117056 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:57.117063 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:57.117120 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:57.122405 (+ 5285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.124953 (+ 2548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.124975 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.124981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.125183 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.125755 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.125760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.125763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.126300 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:57.126609 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:57.130706 (+ 4097us) server_negotiation.cc:299] Negotiation successful
0807 03:01:57.131042 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 03:01:57.098762 1037 master_runner.cc:186] Error getting master registration for 127.1.121.125:45405: OK, instance_id { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" instance_seqno: 1786071716350673 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:01:57.143685 1037 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 03:01:57.146209 1181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:01:57.146462 1181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:01:57.149806 1024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1: No bootstrap required, opened a new log
W20260807 03:01:57.163439 1067 tablet.cc:2369] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:01:57.170373 1093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [term 1 FOLLOWER]: Refusing update from remote peer 3bc6671e0b944fd1b64ae65332362bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:01:57.171689 1024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } }
I20260807 03:01:57.172040 1164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:01:57.174798 1024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cf39412cdc4a91b2360d20d371dcb1, State: Initialized, Role: FOLLOWER
I20260807 03:01:57.175386 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:57.101180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.1.121.124:39367)
0807 03:01:57.101364 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.101370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.132878 (+ 31508us) server_negotiation.cc:407] Connection header received
0807 03:01:57.137615 (+ 4737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.137624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.137758 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.137845 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.147729 (+ 9884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.149179 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.158385 (+ 9206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.159388 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.159556 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:57.165015 (+ 5459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:57.165022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:57.165026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:57.165078 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:57.171139 (+ 6061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.172235 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.172241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.172245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.172424 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.172947 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.172954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.172959 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.173412 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:57.173735 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:57.174786 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0807 03:01:57.175020 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:01:57.175657 1024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } }
I20260807 03:01:57.184324 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:57.105121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.1.121.125:45405)
0807 03:01:57.105340 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.105348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.130425 (+ 25077us) server_negotiation.cc:407] Connection header received
0807 03:01:57.133031 (+ 2606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.133040 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.133155 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.133244 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.143967 (+ 10723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.145575 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.152391 (+ 6816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:57.153467 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:57.153614 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:57.161435 (+ 7821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:01:57.161442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:01:57.161446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:01:57.161494 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:01:57.176621 (+ 15127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.177996 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.178005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.178011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.178368 (+ 357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:01:57.179681 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:01:57.179689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:01:57.179695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:01:57.180652 (+ 957us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:01:57.181438 (+ 786us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:57.182931 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0807 03:01:57.183455 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:01:57.190738 993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bc6671e0b944fd1b64ae65332362bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "50cf39412cdc4a91b2360d20d371dcb1"
I20260807 03:01:57.191251 992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:01:57.190706 1024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:01:57.188483 1189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.193363 1189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:01:57.196349 992 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [term 1 FOLLOWER]: Refusing update from remote peer 3bc6671e0b944fd1b64ae65332362bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:01:57.199975 993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bc6671e0b944fd1b64ae65332362bac in term 1.
I20260807 03:01:57.200496 1164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:01:57.210594 1154 sys_catalog.cc:455] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc6671e0b944fd1b64ae65332362bac. Latest consensus state: current_term: 1 leader_uuid: "3bc6671e0b944fd1b64ae65332362bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.211014 1154 sys_catalog.cc:458] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:01:57.214586 1185 mvcc.cc:205] Tried to move back new op lower bound from 7315749753517592576 to 7315749753127079936. Current Snapshot: MvccSnapshot[applied={T|T < 7315749753517592576}]
I20260807 03:01:57.217828 1190 mvcc.cc:205] Tried to move back new op lower bound from 7315749753517592576 to 7315749753127079936. Current Snapshot: MvccSnapshot[applied={T|T < 7315749753517592576}]
I20260807 03:01:57.221822 1136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bc6671e0b944fd1b64ae65332362bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.222445 1136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:57.224221 1154 sys_catalog.cc:455] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bc6671e0b944fd1b64ae65332362bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.224582 1154 sys_catalog.cc:458] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:01:57.230805 1182 catalog_manager.cc:1347] Generated new cluster ID: 9c5a56b50e7843c3b6682aca875fd541
I20260807 03:01:57.230973 1182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:01:57.241791 1189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc6671e0b944fd1b64ae65332362bac. Latest consensus state: current_term: 1 leader_uuid: "3bc6671e0b944fd1b64ae65332362bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.242129 1189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:01:57.244395 1164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bc6671e0b944fd1b64ae65332362bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.244663 1164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac [sys.catalog]: This master's current role is: LEADER
I20260807 03:01:57.247876 1189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bc6671e0b944fd1b64ae65332362bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc6671e0b944fd1b64ae65332362bac" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 36041 } } peers { permanent_uuid: "50cf39412cdc4a91b2360d20d371dcb1" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 45405 } } peers { permanent_uuid: "b120e2e3535744c1bc0e4a34e6edaefd" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 39367 } } }
I20260807 03:01:57.248167 1189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:01:57.260974 1182 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:01:57.263087 1182 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:01:57.288015 1212 catalog_manager.cc:1259] Loaded cluster ID: 9c5a56b50e7843c3b6682aca875fd541
I20260807 03:01:57.288146 1212 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1: loading cluster ID for follower catalog manager: success
I20260807 03:01:57.295033 1182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bc6671e0b944fd1b64ae65332362bac: Generated new TSK 0
I20260807 03:01:57.297140 1182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:01:57.304028 1212 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1: acquiring CA information for follower catalog manager: success
W20260807 03:01:57.305487 1212 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260807 03:01:57.444703 1509 mini_kdc.cc:348] Time spent kdestroy: real 0.117s user 0.001s sys 0.031s
W20260807 03:01:57.506244 1132 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.491303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.1.121.126:36041)
0807 03:01:57.491907 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.491919 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.502997 (+ 11078us) server_negotiation.cc:407] Connection header received
0807 03:01:57.504434 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.504442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.504547 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.504627 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.505944 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33568: BlockingRecv error: recv got EOF from 127.0.0.1:33568 (error 108)
Metrics: {"server-negotiator.queue_time_us":453}
W20260807 03:01:57.517351 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.503606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.1.121.125:45405)
0807 03:01:57.503801 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.503807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.515451 (+ 11644us) server_negotiation.cc:407] Connection header received
0807 03:01:57.515703 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.515709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.515828 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.515935 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.517110 (+ 1175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34266: BlockingRecv error: recv got EOF from 127.0.0.1:34266 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260807 03:01:57.517974 1124 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.498807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.1.121.124:39367)
0807 03:01:57.499085 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.499095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.507753 (+ 8658us) server_negotiation.cc:407] Connection header received
0807 03:01:57.507918 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.507928 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.508034 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.508167 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.517662 (+ 9495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33876: BlockingRecv error: recv got EOF from 127.0.0.1:33876 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260807 03:01:57.564378 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.551121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.1.121.125:45405)
0807 03:01:57.551325 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.551331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.562017 (+ 10686us) server_negotiation.cc:407] Connection header received
0807 03:01:57.562596 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.562603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.562767 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.562855 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.564032 (+ 1177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34280: BlockingRecv error: recv got EOF from 127.0.0.1:34280 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 03:01:57.613296 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.600449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.1.121.125:45405)
0807 03:01:57.600643 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.600650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.611260 (+ 10610us) server_negotiation.cc:407] Connection header received
0807 03:01:57.611790 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.611798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.611906 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.611976 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.613061 (+ 1085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34284: BlockingRecv error: recv got EOF from 127.0.0.1:34284 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 03:01:57.663565 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.650577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.1.121.125:45405)
0807 03:01:57.650767 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.650773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.661433 (+ 10660us) server_negotiation.cc:407] Connection header received
0807 03:01:57.661967 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.661974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.662122 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.662199 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.663259 (+ 1060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34292: BlockingRecv error: recv got EOF from 127.0.0.1:34292 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 03:01:57.717669 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.704136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.1.121.125:45405)
0807 03:01:57.704350 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.704356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.715440 (+ 11084us) server_negotiation.cc:407] Connection header received
0807 03:01:57.716022 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.716030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.716139 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.716227 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.717423 (+ 1196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34298: BlockingRecv error: recv got EOF from 127.0.0.1:34298 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260807 03:01:57.767495 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.763856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.1.121.125:45405)
0807 03:01:57.764073 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.764083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.765825 (+ 1742us) server_negotiation.cc:407] Connection header received
0807 03:01:57.766018 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.766025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.766151 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.766290 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.767201 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34302: BlockingRecv error: recv got EOF from 127.0.0.1:34302 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 03:01:57.790387 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.787887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.1.121.125:45405)
0807 03:01:57.788097 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.788104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.788657 (+ 553us) server_negotiation.cc:407] Connection header received
0807 03:01:57.788916 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.788922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.789013 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.789119 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.790097 (+ 978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34314: BlockingRecv error: recv got EOF from 127.0.0.1:34314 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260807 03:01:57.833233 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.830925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.1.121.125:45405)
0807 03:01:57.831145 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.831153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.831544 (+ 391us) server_negotiation.cc:407] Connection header received
0807 03:01:57.831820 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.831824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.831896 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.832016 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.832880 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34320: BlockingRecv error: recv got EOF from 127.0.0.1:34320 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260807 03:01:57.903908 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:57.901839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.1.121.125:45405)
0807 03:01:57.902063 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:57.902070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:57.902383 (+ 313us) server_negotiation.cc:407] Connection header received
0807 03:01:57.902574 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:57.902580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:57.902676 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:57.902807 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:57.903683 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34330: BlockingRecv error: recv got EOF from 127.0.0.1:34330 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 03:01:58.037529 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:58.035551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.1.121.125:45405)
0807 03:01:58.035778 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:58.035784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:58.036220 (+ 436us) server_negotiation.cc:407] Connection header received
0807 03:01:58.036380 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:58.036384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:58.036460 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:58.036561 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:58.037336 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34342: BlockingRecv error: recv got EOF from 127.0.0.1:34342 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 03:01:58.156939 1181 catalog_manager.cc:1259] Loaded cluster ID: 9c5a56b50e7843c3b6682aca875fd541
I20260807 03:01:58.157049 1181 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd: loading cluster ID for follower catalog manager: success
I20260807 03:01:58.159837 1181 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd: acquiring CA information for follower catalog manager: success
I20260807 03:01:58.161695 1181 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b120e2e3535744c1bc0e4a34e6edaefd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 03:01:58.299346 1032 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:01:58.297605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.1.121.125:45405)
0807 03:01:58.297814 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:58.297819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:58.298125 (+ 306us) server_negotiation.cc:407] Connection header received
0807 03:01:58.298305 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:58.298309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:58.298383 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:58.298482 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:01:58.299175 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34346: BlockingRecv error: recv got EOF from 127.0.0.1:34346 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:01:58.308171 1212 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50cf39412cdc4a91b2360d20d371dcb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 03:01:58.821501 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:58.815006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.1.121.125:45405)
0807 03:01:58.815533 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:58.815540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:58.815867 (+ 327us) server_negotiation.cc:407] Connection header received
0807 03:01:58.816113 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:58.816117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:58.816330 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:58.816487 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:01:58.816995 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.817804 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.819162 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.819585 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.819737 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:58.820305 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:01:58.820868 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:01:58.821051 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:58.821109 (+ 58us) server_negotiation.cc:299] Negotiation successful
0807 03:01:58.821195 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":264,"threads_started":1}
I20260807 03:01:58.838246 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:58.832617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.1.121.124:39367)
0807 03:01:58.833150 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:58.833155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:58.833185 (+ 30us) server_negotiation.cc:407] Connection header received
0807 03:01:58.833367 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:58.833371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:58.833550 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:58.833734 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:01:58.834106 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.834843 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.836130 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.836546 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.836643 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:58.837167 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:01:58.837725 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:01:58.837838 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:58.837929 (+ 91us) server_negotiation.cc:299] Negotiation successful
0807 03:01:58.837996 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":307,"threads_started":1}
I20260807 03:01:58.856308 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:58.845109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.1.121.124:39367)
0807 03:01:58.845318 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:58.845327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:58.849689 (+ 4362us) server_negotiation.cc:407] Connection header received
0807 03:01:58.850044 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:58.850051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:58.850142 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:58.850247 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:01:58.851287 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.852020 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.854226 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.854728 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.854883 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:58.855379 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:01:58.855514 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:01:58.855601 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:58.855832 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 03:01:58.855940 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:01:58.858839 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:58.849997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.1.121.125:45405)
0807 03:01:58.850477 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:58.850484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:58.851119 (+ 635us) server_negotiation.cc:407] Connection header received
0807 03:01:58.852192 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:58.852196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:58.852263 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:58.852367 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:01:58.852966 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.854034 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.856584 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.857242 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.857370 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:58.858014 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:01:58.858132 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:01:58.858213 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:58.858378 (+ 165us) server_negotiation.cc:299] Negotiation successful
0807 03:01:58.858441 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306}
I20260807 03:01:58.860322 1334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:01:58.845074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.1.121.126:36041)
0807 03:01:58.853426 (+ 8352us) server_negotiation.cc:206] Beginning negotiation
0807 03:01:58.853432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:01:58.853452 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:01:58.853567 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:01:58.853572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:01:58.853798 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:01:58.854016 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:01:58.854734 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.855723 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.858103 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:01:58.858739 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:01:58.858836 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:01:58.859411 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:01:58.859789 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:01:58.859929 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 03:01:58.860054 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 03:01:58.860123 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8223,"thread_start_us":8115,"threads_started":1}
I20260807 03:01:58.865406 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 830
I20260807 03:01:58.866473 917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:58.987600 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 830
I20260807 03:01:59.019695 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 931
I20260807 03:01:59.021003 1019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:59.131049 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 931
I20260807 03:01:59.160218 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 1037
I20260807 03:01:59.161513 1116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:01:59.271646 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 1037
2026-08-07T03:01:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5672 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:01:59 dist-test-slave-3h5h krb5kdc[1344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:01:59 dist-test-slave-3h5h krb5kdc[1344](info): set up 2 sockets
Aug 07 03:01:59 dist-test-slave-3h5h krb5kdc[1344](info): commencing operation
krb5kdc: starting...
W20260807 03:02:00.664290 1509 mini_kdc.cc:121] Time spent starting KDC: real 1.342s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/test-user.keytab.
Aug 07 03:02:00 dist-test-slave-3h5h krb5kdc[1344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.121.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
Entry for principal kudu/127.1.121.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab.
I20260807 03:02:01.028589 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:42481
--webserver_interface=127.1.121.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:42481,127.1.121.125:33985,127.1.121.124:42483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:02:01.401938 1360 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:02:01.402386 1360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:02:01.402529 1360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:02:01.402681 1360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:02:01.411000 1360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:02:01.411144 1360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:02:01.411271 1360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:02:01.411360 1360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:02:01.411469 1360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:02:01.421886 1360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:42481,127.1.121.125:33985,127.1.121.124:42483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.126.keytab
--principal=kudu/127.1.121.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.126:42481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:02:01.423296 1360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:02:01.424777 1360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:02:01.438601 1366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:02:01.438648 1368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:02:01.438697 1370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:02:01.439607 1360 server_base.cc:1034] running on GCE node
I20260807 03:02:01.440459 1360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:02:01.442072 1360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:02:01.443346 1360 hybrid_clock.cc:648] HybridClock initialized: now 1786071721443279 us; error 36 us; skew 500 ppm
I20260807 03:02:01.443701 1360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:02:01 dist-test-slave-3h5h krb5kdc[1344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:02:01.453137 1360 init.cc:377] Logged in from keytab as kudu/127.1.121.126@KRBTEST.COM (short username kudu)
I20260807 03:02:01.454792 1360 webserver.cc:460] Webserver started at http://127.1.121.126:43917/ using document root <none> and password file <none>
I20260807 03:02:01.455618 1360 fs_manager.cc:359] Metadata directory not provided
I20260807 03:02:01.455719 1360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:02:01.456018 1360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:02:01.459419 1360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/data/instance:
uuid: "840847fcf0a945fe8ef51f8144b62f32"
format_stamp: "Formatted at 2026-08-07 03:02:01 on dist-test-slave-3h5h"
server_key: "ee12486254d41dde9d00e15e4ac055da"
server_key_iv: "67122a580ca92713a53794cde333ef95"
server_key_version: "encryptionkey@0"
I20260807 03:02:01.460201 1360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/wal/instance:
uuid: "840847fcf0a945fe8ef51f8144b62f32"
format_stamp: "Formatted at 2026-08-07 03:02:01 on dist-test-slave-3h5h"
server_key: "ee12486254d41dde9d00e15e4ac055da"
server_key_iv: "67122a580ca92713a53794cde333ef95"
server_key_version: "encryptionkey@0"
I20260807 03:02:01.466009 1360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260807 03:02:01.469878 1381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:02:01.471274 1360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:02:01.471427 1360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/wal
uuid: "840847fcf0a945fe8ef51f8144b62f32"
format_stamp: "Formatted at 2026-08-07 03:02:01 on dist-test-slave-3h5h"
server_key: "ee12486254d41dde9d00e15e4ac055da"
server_key_iv: "67122a580ca92713a53794cde333ef95"
server_key_version: "encryptionkey@0"
I20260807 03:02:01.471614 1360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:02:01.491796 1360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:02:01.496130 1360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:02:01.496376 1360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:02:01.515904 1360 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.126:42481
I20260807 03:02:01.515944 1449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.126:42481 every 8 connection(s)
I20260807 03:02:01.517387 1360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/data/info.pb
I20260807 03:02:01.521550 1450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:02:01.526629 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 1360
I20260807 03:02:01.526839 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-0/wal/instance
I20260807 03:02:01.527333 1509 external_mini_cluster.cc:1428] Setting key c43862487efe37f4b72acb7460ea7ff0
I20260807 03:02:01.528371 1450 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:02:01 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:02:01.551415 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.531785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.1.121.126:42481)
0807 03:02:01.532701 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:01.532722 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:01.533238 (+ 516us) server_negotiation.cc:407] Connection header received
0807 03:02:01.535303 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:01.535309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:01.535548 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:01.535755 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:01.537132 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:01.538337 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:01.539758 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:01.540319 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:01.540542 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:01.544045 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:01.544053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:01.544065 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:01.544093 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:01.546930 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:01.547661 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:01.547669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:01.547673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:01.547813 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:01.548232 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:01.548236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:01.548238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:01.548725 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:01.548987 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:01.549522 (+ 535us) server_negotiation.cc:299] Negotiation successful
0807 03:02:01.549881 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":434,"threads_started":1}
I20260807 03:02:01.551378 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.531792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42481 (local address 127.0.0.1:52192)
0807 03:02:01.533064 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.533105 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:02:01.534924 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:02:01.535780 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:02:01.535786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:02:01.536508 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:02:01.536906 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:01.536926 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:01.538505 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:01.538513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:01.539606 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:01.539618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:01.540575 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:01.540582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:01.540756 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:01.541545 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:02:01.541568 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:02:01.543814 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:02:01.547069 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:01.547082 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:01.547096 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:01.547511 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:01.547936 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:01.547942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:01.547945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:01.548100 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:01.548881 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:02:01.548888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:02:01.549148 (+ 260us) client_negotiation.cc:769] Sending connection context
0807 03:02:01.549398 (+ 250us) client_negotiation.cc:240] Negotiation successful
0807 03:02:01.549990 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":139,"spinlock_wait_cycles":70016,"thread_start_us":798,"threads_started":1}
I20260807 03:02:01.554817 1450 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:02:01.556097 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.555680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48068)
0807 03:02:01.555859 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.555977 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 03:02:01.556469 1385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
I20260807 03:02:01.558246 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.557835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48082)
0807 03:02:01.558022 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.558107 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 03:02:01.558476 1450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:33985: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.121.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab.
Entry for principal kudu/127.1.121.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab.
I20260807 03:02:01.607919 1450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } attempt: 1
I20260807 03:02:01.608511 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.125:33985
--webserver_interface=127.1.121.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab
--principal=kudu/127.1.121.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:42481,127.1.121.125:33985,127.1.121.124:42483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:02:01.609593 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.608920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48090)
0807 03:02:01.609199 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.609398 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 03:02:01.610834 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.610362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48094)
0807 03:02:01.610550 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.610652 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 03:02:01.611047 1450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:33985: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
I20260807 03:02:01.679286 1450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } attempt: 2
I20260807 03:02:01.680644 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.680069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48106)
0807 03:02:01.680397 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.680499 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260807 03:02:01.681792 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.681442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48108)
0807 03:02:01.681591 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.681658 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 03:02:01.682044 1450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:33985: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
I20260807 03:02:01.773316 1450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } attempt: 3
I20260807 03:02:01.774631 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.774149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48122)
0807 03:02:01.774359 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.774494 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 03:02:01.775703 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.775337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48124)
0807 03:02:01.775465 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.775566 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 03:02:01.775974 1450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:33985: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
I20260807 03:02:01.919229 1450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } attempt: 4
I20260807 03:02:01.920537 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.919989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48132)
0807 03:02:01.920303 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.920396 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260807 03:02:01.921706 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:01.921314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48136)
0807 03:02:01.921496 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:01.921579 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 03:02:01.922027 1450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.125:33985: Network error: Client connection negotiation failed: client connection to 127.1.121.125:33985: connect: Connection refused (error 111)
W20260807 03:02:01.967263 1459 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:02:01.967620 1459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:02:01.967685 1459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:02:01.967762 1459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:02:01.975356 1459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:02:01.975456 1459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:02:01.975517 1459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:02:01.975570 1459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:02:01.975626 1459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:02:01.985736 1459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:42481,127.1.121.125:33985,127.1.121.124:42483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.125.keytab
--principal=kudu/127.1.121.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.125:33985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.121.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:02:01.987079 1459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:02:01.988400 1459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:02:02.001013 1465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:02:02.001046 1468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:02:02.001078 1466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:02:02.001991 1459 server_base.cc:1034] running on GCE node
I20260807 03:02:02.002955 1459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:02:02.004613 1459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:02:02.005978 1459 hybrid_clock.cc:648] HybridClock initialized: now 1786071722005830 us; error 109 us; skew 500 ppm
I20260807 03:02:02.006350 1459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:02:02.009801 1459 init.cc:377] Logged in from keytab as kudu/127.1.121.125@KRBTEST.COM (short username kudu)
I20260807 03:02:02.011264 1459 webserver.cc:460] Webserver started at http://127.1.121.125:38855/ using document root <none> and password file <none>
I20260807 03:02:02.011950 1459 fs_manager.cc:359] Metadata directory not provided
I20260807 03:02:02.012040 1459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:02:02.012331 1459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:02:02.015542 1459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/data/instance:
uuid: "209cd7a1ed36414cade08ba909b2bbae"
format_stamp: "Formatted at 2026-08-07 03:02:02 on dist-test-slave-3h5h"
server_key: "30d0e3401fefdf745da0813901ae0530"
server_key_iv: "0b1180680c0d04f93c10015b497b4db8"
server_key_version: "encryptionkey@0"
I20260807 03:02:02.016319 1459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/wal/instance:
uuid: "209cd7a1ed36414cade08ba909b2bbae"
format_stamp: "Formatted at 2026-08-07 03:02:02 on dist-test-slave-3h5h"
server_key: "30d0e3401fefdf745da0813901ae0530"
server_key_iv: "0b1180680c0d04f93c10015b497b4db8"
server_key_version: "encryptionkey@0"
I20260807 03:02:02.021886 1459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 03:02:02.025758 1478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:02:02.027211 1459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:02:02.027434 1459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/wal
uuid: "209cd7a1ed36414cade08ba909b2bbae"
format_stamp: "Formatted at 2026-08-07 03:02:02 on dist-test-slave-3h5h"
server_key: "30d0e3401fefdf745da0813901ae0530"
server_key_iv: "0b1180680c0d04f93c10015b497b4db8"
server_key_version: "encryptionkey@0"
I20260807 03:02:02.027678 1459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:02:02.054584 1459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:02:02.059291 1459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:02:02.059597 1459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:02:02.078311 1459 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.125:33985
I20260807 03:02:02.078325 1562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.125:33985 every 8 connection(s)
I20260807 03:02:02.079804 1459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/data/info.pb
I20260807 03:02:02.083706 1563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:02:02.084748 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 1459
I20260807 03:02:02.084993 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-1/wal/instance
I20260807 03:02:02.085575 1509 external_mini_cluster.cc:1428] Setting key 1afac96a35c5f55e778aab132b842f1a
I20260807 03:02:02.094072 1563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:02:02.118567 1565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.097694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.1.121.126:42481)
0807 03:02:02.098278 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.098294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.099303 (+ 1009us) server_negotiation.cc:407] Connection header received
0807 03:02:02.102227 (+ 2924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.102235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.102338 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.102473 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.104022 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.105340 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.106962 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.107722 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.107872 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.111938 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.111943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.111946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.111985 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.114996 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.116079 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.116083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.116085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.116214 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.116710 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.116713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.116715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.117051 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.117178 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.118118 (+ 940us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.118291 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":293,"threads_started":1}
I20260807 03:02:02.120236 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.098249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42481 (local address 127.0.0.1:52194)
0807 03:02:02.099096 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.099149 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 03:02:02.101916 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:02:02.102677 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:02:02.102685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:02:02.103241 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:02:02.103794 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.103810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.105510 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.105514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.106797 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.106810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.107940 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.107944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.108142 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.108986 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:02:02.109008 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:02:02.111676 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:02:02.115187 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.115199 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.115220 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.115954 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.116326 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.116334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.116340 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.116582 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.117186 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:02:02.117198 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:02:02.117724 (+ 526us) client_negotiation.cc:769] Sending connection context
0807 03:02:02.118059 (+ 335us) client_negotiation.cc:240] Negotiation successful
0807 03:02:02.118650 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":306,"threads_started":1}
I20260807 03:02:02.122284 1563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.121.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.121.124@KRBTEST.COM" created.
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 03:02:02.138202 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.123194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48146)
0807 03:02:02.123393 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.123413 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:02:02.123523 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:02:02.124598 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:02:02.124601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:02:02.125116 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:02:02.125278 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.125286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.126671 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.126675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.127967 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.127985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.128878 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.128888 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.129105 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.129733 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:02:02.129761 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:02:02.132409 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:02:02.135609 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.135615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.135619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.135979 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.136352 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.136357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.136360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.136503 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.137178 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:02:02.137183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:02:02.137336 (+ 153us) client_negotiation.cc:769] Sending connection context
0807 03:02:02.137476 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 03:02:02.137748 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 03:02:02.138236 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.123365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.1.121.125:33985)
0807 03:02:02.124170 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.124180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.124215 (+ 35us) server_negotiation.cc:407] Connection header received
0807 03:02:02.124272 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.124277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.124449 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.124563 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.125444 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.126534 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.128120 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.128670 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.128768 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.132636 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.132649 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.132662 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.132694 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.135472 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.136125 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.136133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.136136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.136242 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.136610 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.136616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.136619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.137046 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.137270 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.137645 (+ 375us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.137857 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":423,"threads_started":1}
I20260807 03:02:02.140980 1563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:02:02.142222 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.141694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:34962)
0807 03:02:02.141888 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.142024 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 03:02:02.142464 1485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
I20260807 03:02:02.144057 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.143695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:34972)
0807 03:02:02.143851 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.143939 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 03:02:02.144313 1563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:42483: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.121.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab.
Entry for principal kudu/127.1.121.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab.
I20260807 03:02:02.168896 1509 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska5bS7K/build/asan/bin/kudu
/tmp/dist-test-taska5bS7K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.124:42483
--webserver_interface=127.1.121.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab
--principal=kudu/127.1.121.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.121.84:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.126:42481,127.1.121.125:33985,127.1.121.124:42483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:02:02.193719 1563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } attempt: 1
I20260807 03:02:02.195176 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.194687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:34984)
0807 03:02:02.194919 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.195026 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 03:02:02.196187 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.195837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:34992)
0807 03:02:02.196011 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.196084 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 03:02:02.196399 1563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:42483: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
I20260807 03:02:02.221271 1450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } attempt: 5
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 03:02:02.236433 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.222356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.1.121.125:33985)
0807 03:02:02.222576 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.222582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.222605 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:02:02.222685 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.222690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.222792 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.222902 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.223696 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.224894 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.226031 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.226731 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.226875 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.230173 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.230180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.230185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.230225 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.233472 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.234430 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.234434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.234436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.234552 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.235026 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.235030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.235031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.235313 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.235424 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.236017 (+ 593us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.236208 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:02:02.236776 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.222059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48156)
0807 03:02:02.222312 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.222334 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:02:02.222477 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:02:02.222952 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:02:02.222955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:02:02.223342 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:02:02.223520 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.223528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.225023 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.225027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.225872 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.225890 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.226875 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.226879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.227021 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.227669 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:02:02.227693 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:02:02.229966 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:02:02.233622 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.233630 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.233635 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.234295 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.234680 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.234687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.234694 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.234901 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.235461 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:02:02.235468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:02:02.235682 (+ 214us) client_negotiation.cc:769] Sending connection context
0807 03:02:02.235938 (+ 256us) client_negotiation.cc:240] Negotiation successful
0807 03:02:02.236292 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 03:02:02.237985 1450 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:02:02.239051 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.238675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:34994)
0807 03:02:02.238841 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.238917 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 03:02:02.239964 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.239672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35000)
0807 03:02:02.239807 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.239870 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 03:02:02.240244 1450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:42483: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
I20260807 03:02:02.264699 1563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } attempt: 2
I20260807 03:02:02.266047 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.265587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35004)
0807 03:02:02.265792 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.265887 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 03:02:02.267046 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.266752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35006)
0807 03:02:02.266878 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.266941 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 03:02:02.267287 1563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:42483: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
I20260807 03:02:02.291476 1450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } attempt: 1
I20260807 03:02:02.292748 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.292247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35016)
0807 03:02:02.292471 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.292565 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 03:02:02.293824 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.293466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35018)
0807 03:02:02.293616 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.293717 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:02:02.294102 1450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:42483: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
I20260807 03:02:02.358532 1563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } attempt: 3
I20260807 03:02:02.359786 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.359311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35024)
0807 03:02:02.359514 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.359610 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 03:02:02.361052 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.360565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35036)
0807 03:02:02.360800 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.360939 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260807 03:02:02.361297 1563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:42483: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
I20260807 03:02:02.362324 1450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } attempt: 2
I20260807 03:02:02.363301 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.362858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35044)
0807 03:02:02.363080 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.363155 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 03:02:02.364253 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.363926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35046)
0807 03:02:02.364072 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.364147 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:02:02.364470 1450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:42483: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
I20260807 03:02:02.470707 1450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } attempt: 3
I20260807 03:02:02.471849 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.471331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35054)
0807 03:02:02.471563 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.471657 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 03:02:02.473022 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.472587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35068)
0807 03:02:02.472770 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.472916 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260807 03:02:02.473241 1450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:42483: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
I20260807 03:02:02.504544 1563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } attempt: 4
I20260807 03:02:02.505908 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.505386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35082)
0807 03:02:02.505631 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.505743 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260807 03:02:02.506943 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.506606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35084)
0807 03:02:02.506778 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.506843 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 03:02:02.507144 1563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.121.124:42483: Network error: Client connection negotiation failed: client connection to 127.1.121.124:42483: connect: Connection refused (error 111)
W20260807 03:02:02.539427 1571 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:02:02.539809 1571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:02:02.539944 1571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:02:02.540184 1571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:02:02.547626 1571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:02:02.547752 1571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:02:02.547887 1571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:02:02.547986 1571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:02:02.548097 1571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:02:02.558049 1571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.84:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.126:42481,127.1.121.125:33985,127.1.121.124:42483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/krb5kdc/kudu_127.1.121.124.keytab
--principal=kudu/127.1.121.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.124:42483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.121.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:02:02.559428 1571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:02:02.560812 1571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:02:02.573165 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:02:02.573228 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:02:02.573252 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:02:02.574276 1571 server_base.cc:1034] running on GCE node
I20260807 03:02:02.575129 1571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:02:02.576789 1571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:02:02.578106 1571 hybrid_clock.cc:648] HybridClock initialized: now 1786071722578033 us; error 36 us; skew 500 ppm
I20260807 03:02:02.578469 1571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:02:02.581784 1571 init.cc:377] Logged in from keytab as kudu/127.1.121.124@KRBTEST.COM (short username kudu)
I20260807 03:02:02.583323 1571 webserver.cc:460] Webserver started at http://127.1.121.124:36851/ using document root <none> and password file <none>
I20260807 03:02:02.584061 1571 fs_manager.cc:359] Metadata directory not provided
I20260807 03:02:02.584208 1571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:02:02.584548 1571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:02:02.587752 1571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/data/instance:
uuid: "eb7c829f5075450399a3c52428c96983"
format_stamp: "Formatted at 2026-08-07 03:02:02 on dist-test-slave-3h5h"
server_key: "b3436685a200cf4e06ca5b0ae67d004c"
server_key_iv: "e42a376848d74a4c8f954f6b3fed347a"
server_key_version: "encryptionkey@0"
I20260807 03:02:02.588596 1571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/wal/instance:
uuid: "eb7c829f5075450399a3c52428c96983"
format_stamp: "Formatted at 2026-08-07 03:02:02 on dist-test-slave-3h5h"
server_key: "b3436685a200cf4e06ca5b0ae67d004c"
server_key_iv: "e42a376848d74a4c8f954f6b3fed347a"
server_key_version: "encryptionkey@0"
I20260807 03:02:02.594209 1571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 03:02:02.597728 1593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:02:02.599063 1571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:02:02.599270 1571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/data,/tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/wal
uuid: "eb7c829f5075450399a3c52428c96983"
format_stamp: "Formatted at 2026-08-07 03:02:02 on dist-test-slave-3h5h"
server_key: "b3436685a200cf4e06ca5b0ae67d004c"
server_key_iv: "e42a376848d74a4c8f954f6b3fed347a"
server_key_version: "encryptionkey@0"
I20260807 03:02:02.599495 1571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:02:02.619644 1571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:02:02.624226 1571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:02:02.624527 1571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:02:02.643764 1571 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.124:42483
I20260807 03:02:02.643784 1672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.121.124:42483 every 8 connection(s)
I20260807 03:02:02.645280 1571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/data/info.pb
I20260807 03:02:02.645753 1509 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska5bS7K/build/asan/bin/kudu as pid 1571
I20260807 03:02:02.645977 1509 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska5bS7K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786071323612974-1509-0/minicluster-data/master-2/wal/instance
I20260807 03:02:02.646582 1509 external_mini_cluster.cc:1428] Setting key 99694caf882ae5642ce07120cc572a66
I20260807 03:02:02.649989 1673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:02:02.650457 1450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } attempt: 4
I20260807 03:02:02.659955 1673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071721, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.126@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
I20260807 03:02:02.671440 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.651145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35108)
0807 03:02:02.651436 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.651456 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:02:02.651575 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:02:02.652492 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:02:02.652497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:02:02.653091 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:02:02.653308 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.653321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.655774 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.655780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.657187 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.657202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.658343 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.658348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.658537 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.659282 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:02:02.659309 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:02:02.662824 (+ 3515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:02:02.668543 (+ 5719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.668549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.668553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.669065 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.669525 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.669530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.669533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.669722 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.670384 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:02:02.670389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:02:02.670586 (+ 197us) client_negotiation.cc:769] Sending connection context
0807 03:02:02.670727 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 03:02:02.671037 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260807 03:02:02.672649 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.648519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.1.121.124:42483)
0807 03:02:02.649351 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.649362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.649403 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:02:02.651491 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.651524 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.652112 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.652499 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.653943 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.655466 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.657382 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.658116 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.658335 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.663920 (+ 5585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.663935 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.663940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.663985 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.667748 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.668642 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.668651 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.668659 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.668898 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.669362 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.669368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.669371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.670066 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.670420 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.670861 (+ 441us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.671299 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":623,"thread_start_us":397,"threads_started":1}
I20260807 03:02:02.673085 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.651300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.1.121.124:42483)
0807 03:02:02.652111 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.652119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.652145 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:02:02.652215 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.652221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.652288 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.653471 (+ 1183us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.653784 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.655536 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.657345 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.658135 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.658881 (+ 746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.663183 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.663193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.663214 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.663273 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.668405 (+ 5132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.669219 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.669225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.669229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.669384 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.669857 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.669862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.669865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.670221 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.672154 (+ 1933us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.672286 (+ 132us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.672499 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":669,"thread_start_us":506,"threads_started":1}
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
I20260807 03:02:02.680308 1450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32: Bootstrap starting.
I20260807 03:02:02.682682 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.663149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.126:42481 (local address 127.0.0.1:52210)
0807 03:02:02.664583 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.664639 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 03:02:02.665140 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:02:02.665609 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:02:02.665617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:02:02.666334 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:02:02.666629 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.666639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.668198 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.668204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.669885 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.669898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.671132 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.671138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.671335 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.672020 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:02:02.672055 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:02:02.675405 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:02:02.679380 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.679394 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.679401 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.679972 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.680347 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.680353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.680356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.680538 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.681437 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:02:02.681446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:02:02.681657 (+ 211us) client_negotiation.cc:769] Sending connection context
0807 03:02:02.681906 (+ 249us) client_negotiation.cc:240] Negotiation successful
0807 03:02:02.682267 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":636,"threads_started":1}
I20260807 03:02:02.682845 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.663001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.1.121.126:42481)
0807 03:02:02.663612 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.663618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.664749 (+ 1131us) server_negotiation.cc:407] Connection header received
0807 03:02:02.665318 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.665325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.665417 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.665558 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.666798 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.667995 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.670050 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.670756 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.670838 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.675640 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.675648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.675655 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.675697 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.679232 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.680100 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.680104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.680106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.680229 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.680688 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.680695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.680700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.681190 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.681356 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.682046 (+ 690us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.682341 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":304,"threads_started":1}
I20260807 03:02:02.684787 1673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:02:02.684957 1450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32: Neither blocks nor log segments found. Creating new log.
I20260807 03:02:02.686443 1450 log.cc:824] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32: Log is configured to *not* fsync() on all Append() calls
I20260807 03:02:02.689514 1450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32: No bootstrap required, opened a new log
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.126@KRBTEST.COM
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 03:02:02.700139 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.685531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.125:33985 (local address 127.0.0.1:48162)
0807 03:02:02.685865 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.685885 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:02:02.685993 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:02:02.686392 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:02:02.686397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:02:02.686881 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:02:02.687096 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.687113 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.688595 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.688603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.690168 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.690187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.690918 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.690925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.691101 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.691751 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:02:02.691777 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:02:02.694000 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:02:02.697534 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.697542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.697546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.698002 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.698410 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.698415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.698420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.698557 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.699144 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:02:02.699149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:02:02.699294 (+ 145us) client_negotiation.cc:769] Sending connection context
0807 03:02:02.699456 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 03:02:02.699763 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260807 03:02:02.700353 1684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.677210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.1.121.126:42481)
0807 03:02:02.678190 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.678196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.680791 (+ 2595us) server_negotiation.cc:407] Connection header received
0807 03:02:02.680974 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.680980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.681063 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.681190 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.682857 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.684078 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.685560 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.686271 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.686405 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.692275 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.692283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.692288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.692330 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.696161 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.696997 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.697003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.697008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.697302 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.697788 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.697794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.697797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.698201 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.698463 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.699688 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.699934 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":262,"threads_started":1}
I20260807 03:02:02.700359 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.685653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.1.121.125:33985)
0807 03:02:02.685852 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.685858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.685937 (+ 79us) server_negotiation.cc:407] Connection header received
0807 03:02:02.686128 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.686135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.686238 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.686343 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.687286 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.688363 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.690326 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.690752 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.690902 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.694204 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.694209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.694212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.694244 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.697347 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.698124 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.698130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.698134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.698293 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.698740 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.698745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.698748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.699010 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.699190 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.699742 (+ 552us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.699972 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 03:02:02.701079 1450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } }
I20260807 03:02:02.701488 1450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840847fcf0a945fe8ef51f8144b62f32, State: Initialized, Role: FOLLOWER
I20260807 03:02:02.701938 1673 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:02:02.702090 1450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } }
I20260807 03:02:02.703576 1688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:02.704373 1688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:02:02.705207 1450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [sys.catalog]: configured and running, proceeding with master startup.
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.124@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.121.125@KRBTEST.COM
I20260807 03:02:02.719686 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.702613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35116)
0807 03:02:02.702834 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.702851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:02:02.702970 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:02:02.703624 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:02:02.703628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:02:02.704109 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:02:02.704335 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.704348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.705769 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.705775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.707184 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.707200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.707875 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.707880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.708122 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.709009 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:02:02.709032 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:02:02.712371 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:02:02.716724 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.716732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.716738 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.717261 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.717735 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.717739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.717743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.717900 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.718627 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:02:02.718633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:02:02.718806 (+ 173us) client_negotiation.cc:769] Sending connection context
0807 03:02:02.718960 (+ 154us) client_negotiation.cc:240] Negotiation successful
0807 03:02:02.719281 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":5}
I20260807 03:02:02.720467 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.702964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.1.121.124:42483)
0807 03:02:02.703188 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.703197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.703221 (+ 24us) server_negotiation.cc:407] Connection header received
0807 03:02:02.703268 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.703272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.703346 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.703458 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.704530 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.705624 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.707338 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.707734 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.707865 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.712620 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.712631 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.712639 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.712689 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.716563 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.717412 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.717417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.717424 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.717593 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.718019 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.718023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.718027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.718422 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.718571 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.720044 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.720235 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":65}
I20260807 03:02:02.722908 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.702485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.1.121.125:33985)
0807 03:02:02.702648 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.702654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.702824 (+ 170us) server_negotiation.cc:407] Connection header received
0807 03:02:02.703293 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.703298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.703374 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.703446 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.705278 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.706200 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.708457 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.709129 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.709275 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.714460 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.714469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.714473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.714522 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.718436 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.719524 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.719531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.719537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.719776 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.720275 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.720279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.720282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.720647 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.720957 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.722098 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.722431 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:02:02.726359 1673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983: Bootstrap starting.
W20260807 03:02:02.727985 1704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:02:02.728180 1704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:02:02.730464 1673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983: Neither blocks nor log segments found. Creating new log.
I20260807 03:02:02.731891 1673 log.cc:824] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983: Log is configured to *not* fsync() on all Append() calls
I20260807 03:02:02.721302 1360 master_runner.cc:186] Error getting master registration for 127.1.121.125:33985: OK, instance_id { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" instance_seqno: 1786071722072611 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:02:02.733615 1360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 03:02:02.734050 1509 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:02:02.737453 1417 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "840847fcf0a945fe8ef51f8144b62f32"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52220
I20260807 03:02:02.737989 1417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:02:02.738261 1417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:02:02.741721 1673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983: No bootstrap required, opened a new log
I20260807 03:02:02.741845 1417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } }
I20260807 03:02:02.743984 1417 leader_election.cc:288] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [CANDIDATE]: Term 1 election: Requested vote from peers 209cd7a1ed36414cade08ba909b2bbae (127.1.121.125:33985), eb7c829f5075450399a3c52428c96983 (127.1.121.124:42483)
W20260807 03:02:02.749413 1389 tablet.cc:2369] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:02:02.756587 1673 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } }
I20260807 03:02:02.757663 1673 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7c829f5075450399a3c52428c96983, State: Initialized, Role: FOLLOWER
I20260807 03:02:02.758383 1673 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } }
I20260807 03:02:02.760047 1713 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:02.761142 1713 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:02:02.761524 1673 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:02:02.761580 1628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "840847fcf0a945fe8ef51f8144b62f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eb7c829f5075450399a3c52428c96983"
I20260807 03:02:02.762120 1628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:02:02.766499 1628 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 840847fcf0a945fe8ef51f8144b62f32 in term 1.
I20260807 03:02:02.767584 1386 leader_election.cc:302] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 840847fcf0a945fe8ef51f8144b62f32, eb7c829f5075450399a3c52428c96983; no voters:
I20260807 03:02:02.768128 1688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:02:02.769034 1688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [term 1 LEADER]: Becoming Leader. State: Replica: 840847fcf0a945fe8ef51f8144b62f32, State: Running, Role: LEADER
I20260807 03:02:02.769817 1688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } }
I20260807 03:02:02.774389 1718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840847fcf0a945fe8ef51f8144b62f32. Latest consensus state: current_term: 1 leader_uuid: "840847fcf0a945fe8ef51f8144b62f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:02.774834 1718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [sys.catalog]: This master's current role is: LEADER
I20260807 03:02:02.776854 1725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:02:02.779978 1725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:02:02.806337 1563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } attempt: 5
I20260807 03:02:02.812444 1628 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [term 1 FOLLOWER]: Refusing update from remote peer 840847fcf0a945fe8ef51f8144b62f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:02:02.813671 1718 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 07 03:02:02 dist-test-slave-3h5h krb5kdc[1344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786071722, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.121.125@KRBTEST.COM for kudu/127.1.121.124@KRBTEST.COM
I20260807 03:02:02.833557 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.806922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.121.124:42483 (local address 127.0.0.1:35122)
0807 03:02:02.807211 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0807 03:02:02.807230 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:02:02.807335 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:02:02.808003 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:02:02.808008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:02:02.808707 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:02:02.808992 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.809008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.813053 (+ 4045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.813058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.814555 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:02:02.814570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.815760 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.815765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:02:02.815998 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.816613 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:02:02.816636 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:02:02.819361 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:02:02.824458 (+ 5097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.824492 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.824500 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.825646 (+ 1146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.826611 (+ 965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:02:02.826645 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:02:02.826653 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:02:02.827002 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:02:02.830057 (+ 3055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:02:02.830066 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:02:02.830418 (+ 352us) client_negotiation.cc:769] Sending connection context
0807 03:02:02.830716 (+ 298us) client_negotiation.cc:240] Negotiation successful
0807 03:02:02.831424 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260807 03:02:02.835080 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.807473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35122 (local address 127.1.121.124:42483)
0807 03:02:02.807677 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.807682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.807699 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:02:02.807761 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.807766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.807843 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.807942 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.809807 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.810903 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.814752 (+ 3849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.815420 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.815550 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.819579 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.819588 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.819596 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.819642 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.824191 (+ 4549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.825988 (+ 1797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.825998 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.826004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.826375 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.827300 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.827308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.827341 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.829782 (+ 2441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.831988 (+ 2206us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.832811 (+ 823us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.834138 (+ 1327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":42}
I20260807 03:02:02.835718 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.786432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.1.121.125:33985)
0807 03:02:02.786656 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.786677 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.793003 (+ 6326us) server_negotiation.cc:407] Connection header received
0807 03:02:02.793296 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.793305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.793421 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.793514 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.800555 (+ 7041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.801709 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.805387 (+ 3678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.806344 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.806474 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.810993 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.811000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.811004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.811050 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.815063 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.825616 (+ 10553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.825623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.825627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.825828 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.826322 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.826328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.826331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.826733 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.827025 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.835109 (+ 8084us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.835357 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:02:02.841807 1563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae: Bootstrap starting.
I20260807 03:02:02.843850 1713 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840847fcf0a945fe8ef51f8144b62f32. Latest consensus state: current_term: 1 leader_uuid: "840847fcf0a945fe8ef51f8144b62f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:02.844188 1713 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:02:02.848497 1718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "840847fcf0a945fe8ef51f8144b62f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:02.848752 1563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae: Neither blocks nor log segments found. Creating new log.
I20260807 03:02:02.848805 1718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [sys.catalog]: This master's current role is: LEADER
I20260807 03:02:02.850142 1563 log.cc:824] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae: Log is configured to *not* fsync() on all Append() calls
I20260807 03:02:02.853192 1725 catalog_manager.cc:1347] Generated new cluster ID: 5d10410579fb45639013161bd3d0052f
I20260807 03:02:02.853299 1725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:02:02.854238 1684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.785870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.1.121.126:42481)
0807 03:02:02.786248 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.786258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.789479 (+ 3221us) server_negotiation.cc:407] Connection header received
0807 03:02:02.790629 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.790636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.790730 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.790813 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.796037 (+ 5224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.797598 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.807430 (+ 9832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.808215 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.808320 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.827889 (+ 19569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.827903 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.827908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.827955 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.832044 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.851351 (+ 19307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.851359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.851363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.851560 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.852136 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.852142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.852145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.852613 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.852919 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.853522 (+ 603us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.853830 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260807 03:02:02.868214 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:02.793355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.1.121.124:42483)
0807 03:02:02.793533 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:02.793539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:02.815859 (+ 22320us) server_negotiation.cc:407] Connection header received
0807 03:02:02.817584 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:02.817590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:02.817687 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:02.817768 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:02.827650 (+ 9882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.829116 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.855150 (+ 26034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:02.855888 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:02.856005 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:02.859907 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:02:02.859914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:02:02.859919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:02:02.859969 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:02:02.863851 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.864773 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.864780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.864784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.864986 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:02:02.865652 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:02:02.865657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:02:02.865661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:02:02.866116 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:02:02.866382 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:02.867501 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0807 03:02:02.867790 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":64}
I20260807 03:02:02.855439 1571 master_runner.cc:186] Error getting master registration for 127.1.121.125:33985: OK, instance_id { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" instance_seqno: 1786071722072611 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:02:02.875428 1571 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 03:02:02.878484 1563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae: No bootstrap required, opened a new log
I20260807 03:02:02.878717 1713 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "840847fcf0a945fe8ef51f8144b62f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:02.878984 1713 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:02:02.887688 1725 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:02:02.890523 1725 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:02:02.898895 1563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } }
I20260807 03:02:02.899682 1563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209cd7a1ed36414cade08ba909b2bbae, State: Initialized, Role: FOLLOWER
I20260807 03:02:02.900871 1563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } }
W20260807 03:02:02.917399 1748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:02:02.917660 1748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:02:02.917532 1756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:02.918063 1756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:02:02.920655 1524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:02:02.924528 1524 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [term 1 FOLLOWER]: Refusing update from remote peer 840847fcf0a945fe8ef51f8144b62f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:02:02.924930 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "840847fcf0a945fe8ef51f8144b62f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "209cd7a1ed36414cade08ba909b2bbae"
I20260807 03:02:02.928311 1525 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 840847fcf0a945fe8ef51f8144b62f32 in term 1.
I20260807 03:02:02.928977 1563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:02:02.929687 1718 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:02:02.971009 1725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32: Generated new TSK 0
I20260807 03:02:02.972514 1725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:02:03.004465 1756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840847fcf0a945fe8ef51f8144b62f32. Latest consensus state: current_term: 1 leader_uuid: "840847fcf0a945fe8ef51f8144b62f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:03.004845 1756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:02:03.005142 1767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "840847fcf0a945fe8ef51f8144b62f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:03.005539 1767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:02:03.009125 1718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "840847fcf0a945fe8ef51f8144b62f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840847fcf0a945fe8ef51f8144b62f32" member_type: VOTER last_known_addr { host: "127.1.121.126" port: 42481 } } peers { permanent_uuid: "209cd7a1ed36414cade08ba909b2bbae" member_type: VOTER last_known_addr { host: "127.1.121.125" port: 33985 } } peers { permanent_uuid: "eb7c829f5075450399a3c52428c96983" member_type: VOTER last_known_addr { host: "127.1.121.124" port: 42483 } } }
I20260807 03:02:03.009473 1718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840847fcf0a945fe8ef51f8144b62f32 [sys.catalog]: This master's current role is: LEADER
I20260807 03:02:03.032469 1781 catalog_manager.cc:1259] Loaded cluster ID: 5d10410579fb45639013161bd3d0052f
I20260807 03:02:03.032686 1781 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae: loading cluster ID for follower catalog manager: success
I20260807 03:02:03.037137 1781 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae: acquiring CA information for follower catalog manager: success
I20260807 03:02:03.040143 1781 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 209cd7a1ed36414cade08ba909b2bbae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 03:02:03.098248 1509 mini_kdc.cc:348] Time spent kdestroy: real 0.109s user 0.005s sys 0.028s
W20260807 03:02:03.147116 1568 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.132617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.1.121.125:33985)
0807 03:02:03.132869 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.132877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.141873 (+ 8996us) server_negotiation.cc:407] Connection header received
0807 03:02:03.144457 (+ 2584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.144465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.144600 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.144709 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.146790 (+ 2081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48192: BlockingRecv error: recv got EOF from 127.0.0.1:48192 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 03:02:03.149616 1684 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.136161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.1.121.126:42481)
0807 03:02:03.136400 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.136407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.142467 (+ 6060us) server_negotiation.cc:407] Connection header received
0807 03:02:03.145031 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.145038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.145149 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.145242 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.149206 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52248: BlockingRecv error: recv got EOF from 127.0.0.1:52248 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260807 03:02:03.153160 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.143132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.1.121.124:42483)
0807 03:02:03.143338 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.143345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.150954 (+ 7609us) server_negotiation.cc:407] Connection header received
0807 03:02:03.151443 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.151453 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.151556 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.151655 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.152790 (+ 1135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35124: BlockingRecv error: recv got EOF from 127.0.0.1:35124 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 03:02:03.208015 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:03.186955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.1.121.125:33985)
0807 03:02:03.187208 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.187217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.198212 (+ 10995us) server_negotiation.cc:407] Connection header received
0807 03:02:03.198574 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.198583 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.198834 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.199069 (+ 235us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:03.200014 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.201385 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.204093 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.204985 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.205124 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:03.206116 (+ 992us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:03.206987 (+ 871us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:03.207139 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:03.207445 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 03:02:03.207559 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260807 03:02:03.255224 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.244233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.1.121.124:42483)
0807 03:02:03.244480 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.244488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.252269 (+ 7781us) server_negotiation.cc:407] Connection header received
0807 03:02:03.253027 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.253035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.253208 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.253311 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.254810 (+ 1499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35130: BlockingRecv error: recv got EOF from 127.0.0.1:35130 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 03:02:03.307983 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:03.290613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.1.121.125:33985)
0807 03:02:03.290867 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.290877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.298739 (+ 7862us) server_negotiation.cc:407] Connection header received
0807 03:02:03.299493 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.299504 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.299703 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.299797 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:03.300860 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.302267 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.304994 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.305795 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.305943 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:03.306773 (+ 830us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:03.307000 (+ 227us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:03.307103 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:03.307422 (+ 319us) server_negotiation.cc:299] Negotiation successful
0807 03:02:03.307539 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
W20260807 03:02:03.354452 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.343979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.1.121.124:42483)
0807 03:02:03.344203 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.344210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.351394 (+ 7184us) server_negotiation.cc:407] Connection header received
0807 03:02:03.351695 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.351702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.351810 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.351947 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.354133 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35140: BlockingRecv error: recv got EOF from 127.0.0.1:35140 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 03:02:03.411985 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:03.394813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.1.121.125:33985)
0807 03:02:03.395084 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.395092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.402902 (+ 7810us) server_negotiation.cc:407] Connection header received
0807 03:02:03.403715 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.403724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.403844 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.403931 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:03.405005 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.406436 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.408881 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.409707 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.409835 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:03.410703 (+ 868us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:03.410883 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:03.411003 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:03.411368 (+ 365us) server_negotiation.cc:299] Negotiation successful
0807 03:02:03.411502 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260807 03:02:03.472414 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.447618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.1.121.124:42483)
0807 03:02:03.447866 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.447874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.469663 (+ 21789us) server_negotiation.cc:407] Connection header received
0807 03:02:03.470243 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.470252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.470395 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.470534 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.472086 (+ 1552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35142: BlockingRecv error: recv got EOF from 127.0.0.1:35142 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 03:02:03.545694 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:03.525567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.1.121.125:33985)
0807 03:02:03.525799 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.525808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.538675 (+ 12867us) server_negotiation.cc:407] Connection header received
0807 03:02:03.539226 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.539233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.539328 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.539394 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:03.540070 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.541270 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.543340 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.544068 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.544163 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:03.544820 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:03.545000 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:03.545068 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:03.545294 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 03:02:03.545394 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260807 03:02:03.559734 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.555590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.1.121.124:42483)
0807 03:02:03.556166 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.556175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.558209 (+ 2034us) server_negotiation.cc:407] Connection header received
0807 03:02:03.558430 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.558435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.558536 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.558653 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.559467 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35146: BlockingRecv error: recv got EOF from 127.0.0.1:35146 (error 108)
Metrics: {"server-negotiator.queue_time_us":374}
I20260807 03:02:03.577298 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:03.571272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.1.121.125:33985)
0807 03:02:03.571535 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.571544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.572087 (+ 543us) server_negotiation.cc:407] Connection header received
0807 03:02:03.572334 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.572338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.572408 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.572516 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:03.572988 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.573783 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.575261 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.575782 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.575927 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:03.576400 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:03.576567 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:03.576661 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:03.576822 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 03:02:03.576942 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260807 03:02:03.586181 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.583524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.1.121.124:42483)
0807 03:02:03.583729 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.583738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.584689 (+ 951us) server_negotiation.cc:407] Connection header received
0807 03:02:03.584962 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.584970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.585059 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.585173 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.585958 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35154: BlockingRecv error: recv got EOF from 127.0.0.1:35154 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:02:03.612614 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:03.606451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.1.121.125:33985)
0807 03:02:03.606715 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.606725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.607191 (+ 466us) server_negotiation.cc:407] Connection header received
0807 03:02:03.607390 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.607395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.607468 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.607548 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:03.608015 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.608791 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.610748 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.611179 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.611326 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:03.611807 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:03.611932 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:03.612024 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:03.612216 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 03:02:03.612301 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 03:02:03.619920 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.618236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.1.121.124:42483)
0807 03:02:03.618490 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.618496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.618697 (+ 201us) server_negotiation.cc:407] Connection header received
0807 03:02:03.618833 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.618837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.618909 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.618990 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.619705 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35164: BlockingRecv error: recv got EOF from 127.0.0.1:35164 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 03:02:03.660817 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:03.655428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.1.121.125:33985)
0807 03:02:03.655723 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.655733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.656056 (+ 323us) server_negotiation.cc:407] Connection header received
0807 03:02:03.656295 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.656299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.656401 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.656506 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:03.656995 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.657703 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.659028 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.659493 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.659662 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:03.660073 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:03.660181 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:03.660279 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:03.660433 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 03:02:03.660519 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 03:02:03.668649 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.666875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.1.121.124:42483)
0807 03:02:03.667084 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.667088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.667454 (+ 366us) server_negotiation.cc:407] Connection header received
0807 03:02:03.667598 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.667602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.667678 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.667787 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.668473 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35166: BlockingRecv error: recv got EOF from 127.0.0.1:35166 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:02:03.741681 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:03.736195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.1.121.125:33985)
0807 03:02:03.736453 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.736463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.736862 (+ 399us) server_negotiation.cc:407] Connection header received
0807 03:02:03.737123 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.737128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.737201 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.737311 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:03.737779 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.738542 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.739931 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.740341 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.740494 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:03.740944 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:03.741065 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:03.741167 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:03.741325 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 03:02:03.741407 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 03:02:03.749012 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.747166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.1.121.124:42483)
0807 03:02:03.747369 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.747373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.747718 (+ 345us) server_negotiation.cc:407] Connection header received
0807 03:02:03.747874 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.747878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.747950 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.748037 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.748741 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35172: BlockingRecv error: recv got EOF from 127.0.0.1:35172 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:02:03.886137 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:03.881192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.1.121.125:33985)
0807 03:02:03.881388 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.881393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.881703 (+ 310us) server_negotiation.cc:407] Connection header received
0807 03:02:03.881890 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.881894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.881967 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.882058 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:03.882447 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.883150 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.884478 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:03.885038 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:03.885135 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:03.885516 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:03.885618 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:03.885703 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:03.885858 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 03:02:03.885924 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260807 03:02:03.894120 1676 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:02:03.892254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.1.121.124:42483)
0807 03:02:03.892489 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:03.892496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:03.892714 (+ 218us) server_negotiation.cc:407] Connection header received
0807 03:02:03.892908 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:03.892912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:03.892986 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:03.893067 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:02:03.893839 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35186: BlockingRecv error: recv got EOF from 127.0.0.1:35186 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 03:02:03.919626 1748 catalog_manager.cc:1259] Loaded cluster ID: 5d10410579fb45639013161bd3d0052f
I20260807 03:02:03.919732 1748 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983: loading cluster ID for follower catalog manager: success
I20260807 03:02:03.922220 1748 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983: acquiring CA information for follower catalog manager: success
I20260807 03:02:03.923957 1748 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb7c829f5075450399a3c52428c96983: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 03:02:04.159637 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:04.154552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.1.121.125:33985)
0807 03:02:04.154868 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:04.154873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:04.155083 (+ 210us) server_negotiation.cc:407] Connection header received
0807 03:02:04.155301 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:04.155306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:04.155378 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:04.155463 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:04.155940 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.156648 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.157985 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.158402 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.158511 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:04.158912 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:04.159026 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:04.159095 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:04.159240 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 03:02:04.159306 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260807 03:02:04.172492 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:04.166426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.1.121.124:42483)
0807 03:02:04.166652 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:04.166660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:04.166987 (+ 327us) server_negotiation.cc:407] Connection header received
0807 03:02:04.167219 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:04.167224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:04.167382 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:04.167508 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:04.167972 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.168699 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.170067 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.170634 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.170814 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:04.171404 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:04.171944 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:04.172064 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:04.172219 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 03:02:04.172288 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 03:02:04.185446 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:04.178721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.1.121.125:33985)
0807 03:02:04.178962 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:04.178971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:04.180460 (+ 1489us) server_negotiation.cc:407] Connection header received
0807 03:02:04.180656 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:04.180662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:04.180745 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:04.180869 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:04.181343 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.182080 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.183533 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.183980 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.184119 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:04.184573 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:04.184690 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:04.184796 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:04.185009 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 03:02:04.185113 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:02:04.191915 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:04.179431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.1.121.124:42483)
0807 03:02:04.179638 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:04.179642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:04.185478 (+ 5836us) server_negotiation.cc:407] Connection header received
0807 03:02:04.185761 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:04.185766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:04.185870 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:04.185939 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:04.186790 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.187925 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.190004 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.190563 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.190696 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:04.191132 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:04.191245 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:04.191360 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:04.191516 (+ 156us) server_negotiation.cc:299] Negotiation successful
0807 03:02:04.191599 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:02:04.193014 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:02:04.178590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.1.121.126:42481)
0807 03:02:04.185934 (+ 7344us) server_negotiation.cc:206] Beginning negotiation
0807 03:02:04.185940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:02:04.185959 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:02:04.186095 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:02:04.186100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:02:04.186307 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:02:04.186512 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:02:04.187230 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.188242 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.190270 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:02:04.190886 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:02:04.190982 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:02:04.191515 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:02:04.192015 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:02:04.192545 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0807 03:02:04.192601 (+ 56us) server_negotiation.cc:299] Negotiation successful
0807 03:02:04.192699 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7222,"thread_start_us":7113,"threads_started":1}
I20260807 03:02:04.197433 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 1360
I20260807 03:02:04.198770 1442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:02:04.313374 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 1360
I20260807 03:02:04.347074 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 1459
I20260807 03:02:04.348294 1556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:02:04.459399 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 1459
I20260807 03:02:04.490684 1509 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taska5bS7K/build/asan/bin/kudu pid 1571
I20260807 03:02:04.491868 1665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:02:04.603094 1509 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska5bS7K/build/asan/bin/kudu with pid 1571
2026-08-07T03:02:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5329 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (11002 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (400900 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST